The performance gains from Nvidia's Hopper to Blackwell GPUs come from increased size and power, not efficiency. This signals a potential scaling limit, creating an opportunity for radically new hardware primitives and neural network architectures beyond today's matrix-multiplication-centric models.

The core architectural bet for Cerebras was that incremental improvements on an existing design (like a GPU) yield minimal gains because the incumbent has already optimized it. To achieve a step-change in performance, a fundamentally different approach is required, leading them to their massive, wafer-scale chip design.

NVIDIA's approach requires connecting thousands of Grok chips, creating latency bottlenecks. Cerebras's CEO argues its single, integrated wafer-scale system avoids this "interconnect tax," offering superior memory bandwidth and performance for massive models by eliminating the wiring between thousands of tiny chips.

The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.

SambaNova's SN40 rack outperforms a 140-kilowatt NVIDIA GPU rack with just 10 kilowatts and air cooling. This allows running trillion-parameter models in a single rack, dramatically reducing footprint, power consumption, and the need for specialized liquid-cooled data centers.

Instead of running an entire inference task on a single GPU, the next efficiency leap will come from breaking it down. Tasks like prefill, attention, and feed-forward networks will be routed to specialized chips, such as SRAM-based accelerators, that are best suited for each job, dramatically improving performance and ROI.

Andrew Feldman, CEO of competitor Cerebras, argues their single wafer-scale chip is superior for large AI models. He contends that connecting thousands of smaller GPUs, as Nvidia does, introduces significant latency from physical wiring that negates on-paper performance specs, creating a fundamental bottleneck.

SambaNova's architecture is optimized for inference by treating it as a data movement challenge rather than a raw compute problem. By designing for efficient data flow and communication between memory and compute units, they achieve 5-10x performance improvements over traditional GPUs.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.