Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

The biggest performance breakthroughs in AI are not from isolated improvements in hardware, software, or models. They come from co-designing all three layers simultaneously, turning multiplicative 8x gains into exponential 100x gains, a concept Dylan Patel emphasizes as the key to leapfrogging innovation.

Related Insights

A 10x increase in compute may only yield a one-tier improvement in model performance. This appears inefficient but can be the difference between a useless "6-year-old" intelligence and a highly valuable "16-year-old" intelligence, unlocking entirely new economic applications.

To achieve 1000x efficiency, Unconventional AI is abandoning the digital abstraction (bits representing numbers) that has defined computing for 80 years. Instead, they are co-designing hardware and algorithms where the physics of the substrate itself defines the neural network, much like a biological brain.

AI model capabilities follow a predictable, non-linear scaling law: increasing training compute by 10x roughly doubles a model's capabilities. This exponential relationship, rather than an incremental one, is what will drive underappreciated and disruptive advancements across many industries.

Breakthroughs like neural network "pruning" can reduce model size by 90% without losing accuracy, offering a 10x reduction in inference costs. This highlights that algorithmic innovation, not just acquiring more hardware, will be a key competitive vector in the AI race, enabling more output with less energy.

True co-design between AI models and chips is currently impossible due to an "asymmetric design cycle." AI models evolve much faster than chips can be designed. By using AI to drastically speed up chip design, it becomes possible to create a virtuous cycle of co-evolution.

OpenAI is designing its custom chip for flexibility, not just raw performance on current models. The team learned that major 100x efficiency gains come from evolving algorithms (e.g., dense to sparse transformers), so the hardware must be adaptable to these future architectural changes.

The market often misinterprets AI progress as linear. However, a clear 'scaling law' dictates that a tenfold increase in the computing power used to train LLMs results in a twofold capability improvement. This exponential relationship means future advancements will be far more disruptive and surprising than incremental projections suggest.

We often focus solely on model improvements. Steve Newman argues this is too narrow. True impact is a multiplicative function of eight factors: pre-training, post-training, inference compute, agent scaffolding, app design, user aptitude, workflow refactoring, and adoption. All are advancing simultaneously, creating a blistering pace of change.

The current 2-3 year chip design cycle is a major bottleneck for AI progress, as hardware is always chasing outdated software needs. By using AI to slash this timeline, companies can enable a massive expansion of custom chips, optimizing performance for many at-scale software workloads.

Rethinking and rewriting core systems, like DeepMind's distillation infrastructure, is a prerequisite for advancing research. These large software engineering investments unlock new capabilities, leading to dramatic improvements in model performance and understanding of scaling laws.