The physical area a multiplier circuit requires on a chip grows quadratically with the number of bits (e.g., p*q). This non-linear scaling is the fundamental reason why lower-precision formats like FP4 and FP8 offer disproportionately large performance and efficiency gains for AI workloads compared to a linear improvement.

Since the weight matrix in a systolic array is reused many times, it doesn't need to be loaded quickly. Chip designers can use slow, low-bandwidth connections to "trickle feed" the weights, minimizing the required wiring and thus saving precious die area. This prioritizes area efficiency over initial load latency.

Despite massive financial incentives, high-frequency trading firms rarely develop custom ASICs. CZ explains that FPGAs offer the best trade-off between speed and flexibility. Trading algorithms change too frequently, making the long development cycle of custom silicon impractical compared to reprogrammable FPGAs.

Nvidia’s advantage over ASICs like Google's TPU is programmability. While ASICs are limited to Moore's Law's slow annual gains, CUDA enables radical algorithmic changes that create 10-100x performance leaps, as seen in the jump from Hopper to Blackwell.

While purpose-built chips (ASICs) like Google's TPU are efficient, the AI industry is still in an early, experimental phase. GPUs offer the programmability and flexibility needed to develop new algorithms, as ASICs risk being hard-coded for models that quickly become obsolete.

While adding pipeline registers can increase a chip's clock speed, the registers themselves consume significant silicon area. Over-pipelining can lead to a chip where most of the area is dedicated to registers, not useful logic, resulting in lower overall throughput despite the high clock frequency.

OpenAI is designing its custom chip for flexibility, not just raw performance on current models. The team learned that major 100x efficiency gains come from evolving algorithms (e.g., dense to sparse transformers), so the hardware must be adaptable to these future architectural changes.

For a $1B training run, the subsequent inference costs will exceed $1B. A custom ASIC could save over 20% ($200M+), which is enough to fund the chip's tape-out. This shifts the hardware bottleneck from manufacturing cost to development timeline.

The multiplexer (MUX) circuits required to select and move data from a register file to a logic unit can consume significantly more silicon area than the logic unit performing the actual calculation. This illustrates that data movement is a dominant cost, even at the micro-architectural level.