Existing AI chips force a trade-off: high-throughput HBM memory (NVIDIA, Google) has high latency, while low-latency SRAM memory (Grok) has poor throughput. MatX's architecture combines both, putting model weights in fast SRAM and inference data in high-capacity HBM to achieve both low latency and high throughput.

Nvidia’s advantage over ASICs like Google's TPU is programmability. While ASICs are limited to Moore's Law's slow annual gains, CUDA enables radical algorithmic changes that create 10-100x performance leaps, as seen in the jump from Hopper to Blackwell.

AI workloads are limited by memory bandwidth, not capacity. While commodity DRAM offers more bits per wafer, its bandwidth is over an order of magnitude lower than specialized HBM. This speed difference would starve the GPU's compute cores, making the extra capacity useless and creating a massive performance bottleneck.

The necessity of batching stems from a fundamental hardware reality: moving data is far more energy-intensive than computing with it. A single parameter's journey from on-chip SRAM to the multiplier can cost 1000x more energy than the multiplication itself. Batching amortizes this high data movement cost over many computations.

At a high level, a GPU's architecture consists of many replicated, smaller compute units (SMs), each with its own logic and memory. A TPU has a more centralized, coarse-grained design with a few very large, specialized units. One can think of a GPU as a collection of many tiny TPUs tiled across a chip.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

Designing custom AI hardware is a long-term bet. Google's TPU team co-designs chips with ML researchers to anticipate future needs. They aim to build hardware for the models that will be prominent 2-6 years from now, sometimes embedding speculative features that could provide massive speedups if research trends evolve as predicted.

The key metric for AI chips (GPUs/TPUs) is achieving a high percentage of theoretical peak performance (e.g., 70-80%). This concept, known as "mechanical sympathy," is largely absent in the CPU world, where software performance is so inefficient that measuring against peak is considered nonsensical.

For any given hardware, there is a fundamental lower bound on inference latency. This "latency floor" is the time required to load the model's total parameters from memory (e.g., HBM) onto the chip. This process cannot be sped up by reducing batch size or other software tricks.