Human researchers previously drove approximately 7x annual gains in compute efficiency during pre-training. Employing frontier models as automated research tools to guide reinforcement learning and training infrastructure boosts efficiency gains up to 30x per year.

While RL may not provide perfect cross-domain reasoning (e.g., math to code), its key contribution is teaching models 'horizon generalization.' This is the ability to use more tokens productively over a longer period to make progress on a complex task. This meta-skill is a primary driver of recent capability improvements and appears to be doubling every three months.

Reflection achieved high token efficiency on its 500-billion-parameter Beam model by scaling reinforcement learning across 10,500 GPUs and 100 million rollouts. Rather than rewarding only task completion, the training pipeline explicitly rewarded the model for solving tasks using the fewest possible tokens. Combining compute-efficient token generation with token-saving problem solving creates a specialized reasoner that prevents enterprise inference costs from ballooning.

Pre-training on internet text data is hitting a wall. The next major advancements will come from reinforcement learning (RL), where models learn by interacting with simulated environments (like games or fake e-commerce sites). This post-training phase is in its infancy but will soon consume the majority of compute.

While RL is compute-intensive for the amount of signal it extracts, this is its core economic advantage. It allows labs to trade cheap, abundant compute for expensive, scarce human expertise. RL effectively amplifies the value of small, high-quality human-generated datasets, which is crucial when expertise is the bottleneck.

Moonshot overcame the tendency of LLMs to default to sequential reasoning—a problem they call "serial collapse"—by using Parallel Agent Reinforcement Learning (PARL). They forced an orchestrator model to learn parallelization by giving it time and compute budgets that were impossible to meet sequentially, compelling it to delegate tasks.

AI development is inefficiently split into pre-training (optimizing for compression) and RL (optimizing for tasks), where RL often invalidates pre-training metrics. Combining these into a unified, end-to-end learning algorithm focused on final outcomes could yield an order-of-magnitude improvement in training efficiency.

Instead of relying on expensive, omni-purpose frontier models, companies can achieve better performance and lower costs. By creating a Reinforcement Learning (RL) environment specific to their application (e.g., a code editor), they can train smaller, specialized open-source models to excel at a fraction of the cost.

Continuously training a model on private data internalizes concepts, reducing the need for massive context windows and system prompts. This dramatically cuts token consumption for inference compared to RAG-based approaches that re-read documents repeatedly.