Human researchers previously drove approximately 7x annual gains in compute efficiency during pre-training. Employing frontier models as automated research tools to guide reinforcement learning and training infrastructure boosts efficiency gains up to 30x per year.

Contrary to the conventional view, the training phase for advanced AI models, especially those using reinforcement learning, now demands more compute resources for inference tasks than for the actual backpropagation process where the model learns.

The combined effect of advances in pre-training and reinforcement learning (RL) is multiplicative, not additive. A powerful pre-trained model creates a more sophisticated foundation upon which RL can operate, leading to an accelerating feedback loop of capability. This synergy is a key driver behind the rapid improvement of AI models.

Reinforcement learning achieves superhuman results not by inventing alien concepts, but by surfacing and combining rare behaviors that are already possible within a model's vast pre-trained distribution. The goal of pre-training is to make this search for novel solutions more efficient and less random.

The argument that LLMs are just "stochastic parrots" is outdated. Current frontier models are trained via Reinforcement Learning, where the signal is not "did you predict the right token?" but "did you get the right answer?" This is based on complex, often qualitative criteria, pushing models beyond simple statistical correlation.

Pre-training on internet text data is hitting a wall. The next major advancements will come from reinforcement learning (RL), where models learn by interacting with simulated environments (like games or fake e-commerce sites). This post-training phase is in its infancy but will soon consume the majority of compute.

Allocating substantial compute budgets to reinforcement learning rather than pure pre-training trains models to prune unproductive search paths. This produces models that reach correct conclusions in a fraction of the inference time, cutting token latency and infrastructure costs for production workloads.

AI development is inefficiently split into pre-training (optimizing for compression) and RL (optimizing for tasks), where RL often invalidates pre-training metrics. Combining these into a unified, end-to-end learning algorithm focused on final outcomes could yield an order-of-magnitude improvement in training efficiency.

Pre-training's share of compute utilization has sharply declined as teams master exploratory data recipes and discover that filtering low-quality data outperforms ingesting raw volume. Consequently, model advancement is shifting GPU spending toward post-training reinforcement learning.