Since the weight matrix in a systolic array is reused many times, it doesn't need to be loaded quickly. Chip designers can use slow, low-bandwidth connections to "trickle feed" the weights, minimizing the required wiring and thus saving precious die area. This prioritizes area efficiency over initial load latency.

Existing AI chips force a trade-off: high-throughput HBM memory (NVIDIA, Google) has high latency, while low-latency SRAM memory (Grok) has poor throughput. MatX's architecture combines both, putting model weights in fast SRAM and inference data in high-capacity HBM to achieve both low latency and high throughput.

The necessity of batching stems from a fundamental hardware reality: moving data is far more energy-intensive than computing with it. A single parameter's journey from on-chip SRAM to the multiplier can cost 1000x more energy than the multiplication itself. Batching amortizes this high data movement cost over many computations.

At a high level, a GPU's architecture consists of many replicated, smaller compute units (SMs), each with its own logic and memory. A TPU has a more centralized, coarse-grained design with a few very large, specialized units. One can think of a GPU as a collection of many tiny TPUs tiled across a chip.

NVIDIA's approach requires connecting thousands of Grok chips, creating latency bottlenecks. Cerebras's CEO argues its single, integrated wafer-scale system avoids this "interconnect tax," offering superior memory bandwidth and performance for massive models by eliminating the wiring between thousands of tiny chips.

The fundamental primitive for AI chips isn't arbitrary; it's the multiply-accumulate (MAC) operation. This is because it directly maps to the innermost computational loop of matrix multiplication (output += input1 * input2), which is the foundational computation for most neural networks.

While many focus on compute metrics like FLOPS, the primary bottleneck for large AI models is memory bandwidth—the speed of loading weights into the GPU. This single metric is a better indicator of real-world performance from one GPU generation to the next than raw compute power.

The key advantage of larger GPU clusters is their ability to use the memory bandwidth of all GPUs in parallel to load model weights. This massive aggregate bandwidth dramatically reduces memory fetch times, which is a primary latency bottleneck, especially for very large, sparse models.

Today's transformers are optimized for matrix multiplication (MatMul) on GPUs. However, as compute scales to distributed clusters, MatMul may not be the most efficient primitive. Future AI architectures could be drastically different, built on new primitives better suited for large-scale, distributed hardware.