The next wave of AI silicon may pivot from today's compute-heavy architectures to memory-centric ones optimized for inference. This fundamental shift would allow high-performance chips to be produced on older, more accessible 7-14nm manufacturing nodes, disrupting the current dependency on cutting-edge fabs.

World models are algorithmically more intense than language models, pushing computation (flops) much harder relative to memory access. This unique computational pattern will create a market for specialized chips optimized specifically for these workloads, leading to a divergence from the current hardware landscape built for LLMs.

The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.

The GPU architecture is economically optimized for slow AI inference, offering a very low cost per token. However, this efficiency plummets when speed is required, as the cost and power per token increase exponentially, creating a market for alternative architectures in high-speed applications.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

NVIDIA's commitment to programmable GPUs over fixed-function ASICs (like a "transformer chip") is a strategic bet on rapid AI innovation. Since models are evolving so quickly (e.g., hybrid SSM-transformers), a flexible architecture is necessary to capture future algorithmic breakthroughs.

The intense power demands of AI inference will push data centers to adopt the "heterogeneous compute" model from mobile phones. Instead of a single GPU architecture, data centers will use disaggregated, specialized chips for different tasks to maximize power efficiency, creating a post-GPU era.

OpenAI is designing its custom chip for flexibility, not just raw performance on current models. The team learned that major 100x efficiency gains come from evolving algorithms (e.g., dense to sparse transformers), so the hardware must be adaptable to these future architectural changes.

The rise of agent orchestration using specialized, open-source models will drive demand for custom ASICs. Jerry Murdock argues that putting a model on a dedicated chip will be far cheaper and more tunable for specific workloads than using expensive, general-purpose GPUs like Nvidia's, spurring a hardware shift.