The AI inference process involves two distinct phases: "prefill" (reading the prompt, which is compute-bound) and "decode" (writing the response, which is memory-bound). NVIDIA GPUs excel at prefill, while companies like Grok optimize for decode. The Grok-NVIDIA deal signals a future of specialized, complementary hardware rather than one-size-fits-all chips.
Related Insights
The next wave of AI silicon may pivot from today's compute-heavy architectures to memory-centric ones optimized for inference. This fundamental shift would allow high-performance chips to be produced on older, more accessible 7-14nm manufacturing nodes, disrupting the current dependency on cutting-edge fabs.
New AI models are designed to perform well on available, dominant hardware like NVIDIA's GPUs. This creates a self-reinforcing cycle where the incumbent hardware dictates which model architectures succeed, making it difficult for superior but incompatible chip designs to gain traction.
While competitors chased cutting-edge physics, AI chip company Groq used a more conservative process technology but loaded its chip with on-die memory (SRAM). This seemingly less advanced but different architectural choice proved perfectly suited for the "decode" phase of AI inference, a critical bottleneck that led to its licensing deal with NVIDIA.
Nvidia dominates AI because its GPU architecture was perfect for the new, highly parallel workload of AI training. Market leadership isn't just about having the best chip, but about having the right architecture at the moment a new dominant computing task emerges.
Nvidia paid $20 billion for a non-exclusive license from chip startup Groq. This massive price for a non-acquisition signals Nvidia perceived Groq's inference-specialized chip as a significant future competitor in the post-training AI market. The deal neutralizes a threat while absorbing key technology and talent for the next industry battleground.
Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.
Anthropic's choice to purchase Google's TPUs via Broadcom, rather than directly or by designing its own chips, indicates a new phase in the AI hardware market. It highlights the rise of specialized manufacturers as key suppliers, creating a more complex and diversified hardware ecosystem beyond just Nvidia and the major AI labs.
Beyond the simple training-inference binary, Arm's CEO sees a third category: smaller, specialized models for reinforcement learning. These chips will handle both training and inference, acting like 'student teachers' taught by giant foundational models.
The fundamental unit of AI compute has evolved from a silicon chip to a complete, rack-sized system. According to Nvidia's CTO, a single 'GPU' is now an integrated machine that requires a forklift to move, a crucial mindset shift for understanding modern AI infrastructure scale.
The competitive threat from custom ASICs is being neutralized as NVIDIA evolves from a GPU company to an "AI factory" provider. It is now building its own specialized chips (e.g., CPX) for niche workloads, turning the ASIC concept into a feature of its own disaggregated platform rather than an external threat.
