The original playbook of simply scaling parameters and data is now obsolete. Top AI labs have pivoted to heavily designed post-training pipelines, retrieval, tool use, and agent training, acknowledging that raw scaling is insufficient to solve real-world problems.

Unlike traditional ML where models are repeatedly trained on a fixed dataset, each frontier LLM pre-training run uses more compute than ever before. This makes it a one-shot endeavor where success hinges on accurately predicting final performance from smaller-scale experiments using scaling laws.

The Chinchilla scaling law optimizes pre-training compute alone. However, production models must also account for inference costs. By training smaller models on much more data (~100x the Chinchilla optimum), labs create models that are cheaper to run for users, effectively amortizing the higher training cost over the model's lifetime.

When an LLM sees the same document thousands of times, it prioritizes memorizing that specific content to lower its training loss. This gives a false impression of progress, while the model fails to generalize to new, unseen data. A study of Google's T5 corpus found one sentence repeated over 60,000 times.

Contrary to the "more data is better" mantra, scaling with bad data actively degrades model performance. Undeduplicated data makes models "forgetful" and less intelligent over time. You cannot overcome poor data quality simply by adding more compute; better, cleaner data is more effective.

Customizing a base model with proprietary data is only effective if a company possesses a massive corpus. At least 10 billion high-quality tokens are needed *after* aggressive deduplication and filtering. This high threshold means the strategy is only viable for the largest corporations, a much higher bar than most businesses realize.

A study that trained old models on new datasets (and vice versa) found that data improvements accounted for a 9x gain in compute efficiency, while architectural changes only provided a 3x gain. This suggests that data quality, filtering, and synthesis have been the predominant drivers of progress in pre-training, at least at smaller scales.

The current commercial training cadence introduces a dangerous capability overhang. Deploying newly scaled foundation models every six months outpaces the time required to comprehend how existing models function, where their behavioral vulnerabilities lie, or why they fail. As capital requirements jump from tens of millions to billions of dollars, forced pauses between massive training runs could provide a necessary window for researchers to evaluate model bounds rather than blindly stacking parameters.

The rapid, step-change improvements in LLMs are likely slowing down. This is because models have already been trained on most of the available internet, and the compute budget required for each incremental improvement is increasing exponentially to an unsustainable degree. A new architectural breakthrough, not just more data and compute, is needed for the next leap.