The AI inference process involves two distinct phases: "prefill" (reading the prompt, which is compute-bound) and "decode" (writing the response, which is memory-bound). NVIDIA GPUs excel at prefill, while companies like Grok optimize for decode. The Grok-NVIDIA deal signals a future of specialized, complementary hardware rather than one-size-fits-all chips.

Google is abandoning its single-line TPU strategy, now working with both Broadcom and MediaTek on different, specialized TPU designs. This reflects an industry-wide realization that no single chip can be optimal for the diverse and rapidly evolving landscape of AI tasks.

Designing custom AI hardware is a long-term bet. Google's TPU team co-designs chips with ML researchers to anticipate future needs. They aim to build hardware for the models that will be prominent 2-6 years from now, sometimes embedding speculative features that could provide massive speedups if research trends evolve as predicted.

Google successfully trained its top model, Gemini 3 Pro, on its own TPUs, proving a viable alternative to NVIDIA's chips. However, because Google doesn't sell these TPUs, NVIDIA retains its monopoly pricing power over every other company in the market.

The intense power demands of AI inference will push data centers to adopt the "heterogeneous compute" model from mobile phones. Instead of a single GPU architecture, data centers will use disaggregated, specialized chips for different tasks to maximize power efficiency, creating a post-GPU era.

Anthropic's choice to purchase Google's TPUs via Broadcom, rather than directly or by designing its own chips, indicates a new phase in the AI hardware market. It highlights the rise of specialized manufacturers as key suppliers, creating a more complex and diversified hardware ecosystem beyond just Nvidia and the major AI labs.

The inference market is too large to remain monolithic. It will fragment into specialized platforms for different use cases like real-time video, long-running agents, or language models. This specialization will extend to hardware, with high-throughput, low-latency-need tasks (like agents) favoring cheaper AMD/Intel chips over NVIDIA's top GPUs.

Beyond the simple training-inference binary, Arm's CEO sees a third category: smaller, specialized models for reinforcement learning. These chips will handle both training and inference, acting like 'student teachers' taught by giant foundational models.

The narrative of NVIDIA's untouchable dominance is undermined by a critical fact: the world's leading models, including Google's Gemini 3 and Anthropic's Claude 4.5, are primarily trained on Google's TPUs and Amazon's Tranium chips. This proves that viable, high-performance alternatives already exist at the highest level of AI development.