The biggest performance breakthroughs in AI are not from isolated improvements in hardware, software, or models. They come from co-designing all three layers simultaneously, turning multiplicative 8x gains into exponential 100x gains, a concept Dylan Patel emphasizes as the key to leapfrogging innovation.
The TPU vs. GPU debate is a proxy for model architecture. OpenAI's sparse models are co-designed for NVIDIA GPUs, while Google and Anthropic's denser models are optimized for TPUs. Choosing a chip is effectively a long-term bet on a specific architectural path for AI models.
The "CUDA moat" is misunderstood. NVIDIA's true advantage is that major open-source models (e.g., from DeepSeek, Alibaba) are co-designed for its GPUs. This creates a powerful downstream effect where developers must use NVIDIA hardware to run the best available models, regardless of the programming layer.
Specialized AI clouds (NeoClouds) like CoreWeave emerged because hyperscalers' strengths—such as custom networking and security for multi-tenancy—were detrimental to the performance of large-scale, single-tenant AI workloads. This performance gap created a significant market opening.
Jensen Huang strategically allocates GPUs to NeoClouds and new AI labs to prevent a world dominated by a few hyperscalers building their own custom chips (like TPUs). This ensures a diverse customer base and prevents NVIDIA's core products from being commoditized by a handful of powerful buyers.
The critical trade-off in AI is between throughput (cost efficiency via batching) and interactivity (low latency for users). This curve dictates infrastructure, model, and application decisions, determining whether a workload is optimized for cheap batch processing or high-value instant responses.
The current compute crunch isn't just a supply issue. It's because new AI models are so much more capable that they unlock a total addressable market (TAM) of valuable tasks that grows exponentially, far outpacing the linear or geometric growth of compute supply.
Traditional, point-in-time AI benchmarks are useless because the software stack (models, libraries, drivers) updates constantly, with some libraries deploying twice a week. This relentless optimization requires "living" benchmarks that run continuously to remain relevant.
Dylan Patel predicts that while orbital data centers are irrelevant for the next 3-5 years, by 2040 they will be essential. The sheer scale of AI's power demand (terawatts) will make terrestrial power and land the primary bottleneck, forcing new compute deployments into space.
For two decades, silicon chips have been thermally constrained to a power density of about 1 watt per square millimeter. New R&D efforts are finally overcoming this barrier, which could lead to smaller, more powerful chips, despite significant thermal and electrical engineering challenges.
Google isn't betting on a single chip design. It's actively developing three distinct TPU architectures with different partners to avoid being trapped in a "local minima." This hedges against future breakthroughs in model architecture that could render one design obsolete.