Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Emmanuel says the bet was not just compute but finding an objective where compute keeps returning value. He notes scaling laws showed that pouring more compute and data into next-word prediction predictably improved a metric that correlates with many useful tasks, which was rare in practice.

Related Insights

A 10x increase in compute may only yield a one-tier improvement in model performance. This appears inefficient but can be the difference between a useless "6-year-old" intelligence and a highly valuable "16-year-old" intelligence, unlocking entirely new economic applications.

Skeptics continuously argued that transformer models would hit performance walls and remain merely probabilistic token-matchers. Yet Roose emphasizes that, historically, researchers who consistently bet on scaling compute and model size have been validated at every step.

The relationship between computing power and AI model capability is not linear. According to established 'scaling laws,' a tenfold increase in the compute used for training large language models (LLMs) results in roughly a doubling of the model's capabilities, highlighting the immense resources required for incremental progress.

A key surprise in AI development was the non-linear impact of scale. Sebastian Thrun noted that while AI trained on millions of documents is 'fine,' training it on hundreds of billions creates an 'unbelievably smart' system, shocking even its creators and demonstrating data volume as a primary driver of breakthroughs.

Unlike traditional ML where models are repeatedly trained on a fixed dataset, each frontier LLM pre-training run uses more compute than ever before. This makes it a one-shot endeavor where success hinges on accurately predicting final performance from smaller-scale experiments using scaling laws.

The impact of algorithmic progress is so immense that if we trained a model today using the same amount of compute as 2020's GPT-3, the resulting model would likely be more capable than 2023's much larger GPT-4. This highlights that compute isn't the only driver of progress.

A study that trained old models on new datasets (and vice versa) found that data improvements accounted for a 9x gain in compute efficiency, while architectural changes only provided a 3x gain. This suggests that data quality, filtering, and synthesis have been the predominant drivers of progress in pre-training, at least at smaller scales.

The market often misinterprets AI progress as linear. However, a clear 'scaling law' dictates that a tenfold increase in the computing power used to train LLMs results in a twofold capability improvement. This exponential relationship means future advancements will be far more disruptive and surprising than incremental projections suggest.

The rapid, step-change improvements in LLMs are likely slowing down. This is because models have already been trained on most of the available internet, and the compute budget required for each incremental improvement is increasing exponentially to an unsustainable degree. A new architectural breakthrough, not just more data and compute, is needed for the next leap.

Scaling laws do not automatically exist everywhere. Much of the machine learning engineering challenge is finding the specific domain, architecture, and data configuration where adding more compute and data reliably yields better performance.