Significant opportunity exists in re-architecting how AI models work. Instead of building ever-larger single models, the focus is shifting to creating networks of smaller, specialized models that collaborate, which can drastically reduce the cost per token produced.

The AI industry is hitting data limits for training massive, general-purpose models. The next wave of progress will likely come from creating highly specialized models for specific domains, similar to DeepMind's AlphaFold, which can achieve superhuman performance on narrow tasks.

Breakthroughs like neural network "pruning" can reduce model size by 90% without losing accuracy, offering a 10x reduction in inference costs. This highlights that algorithmic innovation, not just acquiring more hardware, will be a key competitive vector in the AI race, enabling more output with less energy.

When designing smaller models, it's inefficient to use limited parameters for memorizing facts that can be looked up. Jeff Dean advocates for focusing a model's capacity on core reasoning abilities and pairing it with a retrieval system. This makes the model more generally useful, as it can access a vast external knowledge base when needed.

Advanced AI architectures will use small, fast, and cheap local models to act as intelligent routers. These models will first analyze a complex request, formulate a plan, and then delegate different sub-tasks to a fleet of more powerful or specialized models, optimizing for cost and performance.

Successful AI models will be small, specialized ones that run efficiently on consumer CPUs at the edge (laptops, phones). This leverages existing hardware (e.g., Apple's M-series chips) and avoids costly cloud GPUs, creating a strategic advantage for companies like Apple.

Current AI models become exponentially more expensive as input size grows (quadratic scaling). New "subquadratic" architectures, however, scale linearly by pre-selecting relevant data. This change could slash compute costs by orders of magnitude, making massive context windows economically viable.

New AI models are moving away from brute-force computation. By selectively focusing on relevant data, much like the human brain indexes memories, they can achieve massive performance gains and cost reductions, overcoming a major bottleneck in current architectures.

The true commercial impact of AI will likely come from small, specialized "micro models" solving boring, high-volume business tasks. While highly valuable, these models are cheap to run and cannot economically justify the current massive capital expenditure on AGI-focused data centers.