What Is a Systolic Array? The Grid Behind Fast Matrix Math

A systolic array is a grid of simple processing elements, each connected only to its immediate neighbors, that pass data through the grid in a fixed, rhythmic pattern — data flows in from the edges, each element performs a small computation and forwards the result onward, and results emerge from the opposite edge once the wave of computation has propagated through. It’s the architecture behind the matrix-multiplication units at the heart of most modern AI accelerators, including Google’s TPUs.

Why matrix multiplication needed its own hardware

Neural network inference and training is overwhelmingly matrix multiplication — layer after layer of multiplying an input matrix by a weight matrix. A general-purpose CPU core handles this the way it handles anything else: fetch operands from memory, multiply, fetch the next operands, multiply again. For matrix multiplication at the scale modern models require, that memory traffic becomes the bottleneck long before the arithmetic does — the chip spends more time moving numbers than doing math with them.

A systolic array attacks that bottleneck directly. Instead of fetching each operand from memory for every operation, data is loaded once at the array’s edge and then reused as it flows from processing element to processing element, with each element performing a multiply-accumulate step and passing partial results to its neighbor. The name comes from the analogy to a heartbeat — data pulses through the grid in synchronized steps, the way blood is pumped through a circulatory system, hence “systolic.”

How the grid actually computes

Picture a 2D grid of small processing elements, each holding one weight value. Input values stream in from one edge (say, the left), and partial sums stream in from another (say, the top). On each clock cycle, every element:

1. Multiplies its stored weight by the input value passing through it
2. Adds that product to the partial sum flowing through it
3. Passes both the input and the updated partial sum to its neighbors

After enough cycles for data to propagate across the full grid, complete matrix products emerge from the far edge. Because each weight stays fixed in place for the duration of the computation and every element only ever talks to its immediate neighbors, the design avoids the long-distance data movement that dominates cost on a general-purpose architecture — memory access, which is disproportionately expensive relative to arithmetic on modern hardware.

Why this fits AI workloads specifically

Systolic arrays are a poor fit for general-purpose computing — branching logic, irregular data access, and varied instruction types don’t map onto a fixed grid of identical elements that all execute the same rhythmic pattern. But that rigidity is exactly what makes them efficient for the narrow, repetitive workload that dominates deep learning: the same multiply-accumulate operation, applied over and over, across a predictable, regular flow of data.

This is a big part of why purpose-built AI accelerators outperform general-purpose processors on training and inference despite having less flexible instruction sets — see CPU vs GPU vs TPU for how that division of labor plays out across an entire system. A systolic array is one specific way to build the matrix-multiply core of such a chip; it’s a design choice within the broader category of application-specific integrated circuits, where hardware is built around one workload’s data flow rather than kept general-purpose.

Systolic arrays vs a general SIMD unit

	General SIMD unit (GPU-style)	Systolic array
Data movement	Operands fetched from memory per operation	Data flows between neighboring elements
Flexibility	Handles varied instructions and workloads	Fixed for one computation pattern, typically matrix multiply
Efficiency on matrix math	Good, with careful memory optimization	Very high — memory traffic is minimized by design
Efficiency on general workloads	Good	Poor — the grid isn’t built for it

Where the idea shows up in silicon

The concept predates the current AI boom by decades — systolic arrays were proposed for signal processing back in the 1970s — but they found their most visible modern application in dedicated matrix-multiply units inside AI accelerator chips. The tradeoff those chips make is the same one that shapes most chiplet and process-node decisions in accelerator design: trade general-purpose flexibility for raw throughput on the one operation that dominates the workload.

Scaling limits and design tradeoffs

Systolic arrays aren’t infinitely scalable in a single pass. A matrix that’s larger than the physical grid has to be broken into tiles that are streamed through the array in sequence, with partial results combined afterward — a technique called tiling that shows up throughout high-performance matrix computation, not just in systolic designs. The size of the physical grid becomes a real design tradeoff: a larger array processes bigger tiles per pass and needs less tiling overhead, but consumes more die area and power, competing with everything else a chip designer wants to fit on the same silicon budget alongside considerations like thermal design power limits. Chipmakers building AI accelerators tune this grid size against the matrix dimensions their target workloads actually use, since a mismatch in either direction wastes either silicon or throughput.

The takeaway

A systolic array is a grid of simple processing elements passing data to their neighbors in a fixed rhythm, built specifically to minimize the memory traffic that dominates matrix multiplication on general-purpose hardware. It’s inflexible outside that one job, which is precisely why it’s fast at it — a trade that makes sense when a workload as narrow and repetitive as neural network math justifies hardware built around its exact data flow.