While purpose-built chips (ASICs) like Google's TPU are efficient, the AI industry is still in an early, experimental phase. GPUs offer the programmability and flexibility needed to develop new algorithms, as ASICs risk being hard-coded for models that quickly become obsolete.

To achieve 1000x efficiency, Unconventional AI is abandoning the digital abstraction (bits representing numbers) that has defined computing for 80 years. Instead, they are co-designing hardware and algorithms where the physics of the substrate itself defines the neural network, much like a biological brain.

The next wave of AI silicon may pivot from today's compute-heavy architectures to memory-centric ones optimized for inference. This fundamental shift would allow high-performance chips to be produced on older, more accessible 7-14nm manufacturing nodes, disrupting the current dependency on cutting-edge fabs.

Top-tier kernels like FlashAttention are co-designed with specific hardware (e.g., H100). This tight coupling makes waiting for future GPUs an impractical strategy. The competitive edge comes from maximizing the performance of available hardware now, even if it means rewriting kernels for each new generation.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

Hyperscalers face a strategic challenge: building massive data centers with current chips (e.g., H100) risks rapid depreciation as far more efficient chips (e.g., GB200) are imminent. This creates a 'pause' as they balance fulfilling current demand against future-proofing their costly infrastructure.

AI progress was expected to stall in 2024-2025 due to hardware limitations on pre-training scaling laws. However, breakthroughs in post-training techniques like reasoning and test-time compute provided a new vector for improvement, bridging the gap until next-generation chips like NVIDIA's Blackwell arrived.

OpenAI is designing its custom chip for flexibility, not just raw performance on current models. The team learned that major 100x efficiency gains come from evolving algorithms (e.g., dense to sparse transformers), so the hardware must be adaptable to these future architectural changes.

Arvind Krishna forecasts a 1000x drop in AI compute costs over five years. This won't just come from better chips (a 10x gain). It will be compounded by new processor architectures (another 10x) and major software optimizations like model compression and quantization (a final 10x).