The AI industry is hitting data limits for training massive, general-purpose models. The next wave of progress will likely come from creating highly specialized models for specific domains, similar to DeepMind's AlphaFold, which can achieve superhuman performance on narrow tasks.

Even a specialized task like coding involves a wide range of human-like interaction: brainstorming, searching, and more. This "AGI-completeness" means a powerful general model with a good "bedside manner" can outperform a narrowly specialized one, complicating the strategy for vertical AI apps.

AI development history shows that complex, hard-coded approaches to intelligence are often superseded by more general, simpler methods that scale more effectively. This "bitter lesson" warns against building brittle solutions that will become obsolete as core models improve.

While previously underwhelming, the latest generation of AI models are now surprisingly effective at highly specialized, low-level coding tasks such as writing GPU shaders. This shows that the "bitter lesson"—that general models scaling beats specialized approaches—applies even in embedded and systems programming.

Today's AI boom is fueled by scaling computation, which is a known engineering challenge. The alternative, embedding nuanced, human-like inductive biases, is far harder as it requires a deep understanding of the problem space. This difficulty gap explains why massive models dominate AI development over more targeted, efficient ones—scaling is simply the more straightforward path.

The history of AI, such as the 2012 AlexNet breakthrough, demonstrates that scaling compute and data on simpler, older algorithms often yields greater advances than designing intricate new ones. This "bitter lesson" suggests prioritizing scalability over algorithmic complexity for future progress.

The "bitter lesson" in AI research posits that methods leveraging massive computation scale better and ultimately win out over approaches that rely on human-designed domain knowledge or clever shortcuts, favoring scale over ingenuity.

By removing all human game data and learning only from self-play, AlphaZero first rediscovered human strategies and then discarded them for superior, 'alien' ones. This showed that relying solely on human data can limit an AI's potential, anchoring it to existing knowledge and cognitive biases.

Just as neural networks replaced hand-crafted features, large generalist models are replacing narrow, task-specific ones. Jeff Dean notes the era of unified models is "really upon us." A single, large model that can generalize across domains like math and language is proving more powerful than bespoke solutions for each, a modern take on the "bitter lesson."