Startups can make big bets on emerging workloads, like LLMs before they were proven. This is a product risk. In contrast, incumbents like Google or NVIDIA must ensure their next chip serves a wide range of existing customers, forcing them to be more conservative and avoid disruptive product bets.

The competitive landscape for AI chips is not a crowded field but a battle between two primary forces: NVIDIA’s integrated system (hardware, software, networking) and Google's TPU. Other players like AMD and Broadcom are effectively a combined secondary challenger offering an open alternative.

Unlike competitors, MatX's ML team conducts fundamental research, training LLMs to validate novel hardware choices. This allows them to safely "cut corners" on industry standards, such as using less precise rounding methods. This deep co-design of model and hardware creates a uniquely efficient product.

Google is abandoning its single-line TPU strategy, now working with both Broadcom and MediaTek on different, specialized TPU designs. This reflects an industry-wide realization that no single chip can be optimal for the diverse and rapidly evolving landscape of AI tasks.

Designing custom AI hardware is a long-term bet. Google's TPU team co-designs chips with ML researchers to anticipate future needs. They aim to build hardware for the models that will be prominent 2-6 years from now, sometimes embedding speculative features that could provide massive speedups if research trends evolve as predicted.

NVIDIA's commitment to CUDA's backward compatibility prevents it from making fundamental changes to its chip architecture. This creates an opportunity for new players like MatX to build chips from a blank slate, optimized purely for modern LLM workloads without being tied to a decade-old programming model.

Anthropic's choice to purchase Google's TPUs via Broadcom, rather than directly or by designing its own chips, indicates a new phase in the AI hardware market. It highlights the rise of specialized manufacturers as key suppliers, creating a more complex and diversified hardware ecosystem beyond just Nvidia and the major AI labs.

The current 2-3 year chip design cycle is a major bottleneck for AI progress, as hardware is always chasing outdated software needs. By using AI to slash this timeline, companies can enable a massive expansion of custom chips, optimizing performance for many at-scale software workloads.

Google created its custom TPU chip not as a long-term strategy, but from an internal crisis. Engineer Jeff Dean calculated that scaling a new speech recognition feature to all Android phones would require doubling Google's entire data center footprint, forcing the company to design a more efficient, custom chip to avoid existential costs.