The combined effect of advances in pre-training and reinforcement learning (RL) is multiplicative, not additive. A powerful pre-trained model creates a more sophisticated foundation upon which RL can operate, leading to an accelerating feedback loop of capability. This synergy is a key driver behind the rapid improvement of AI models.

Reinforcement learning achieves superhuman results not by inventing alien concepts, but by surfacing and combining rare behaviors that are already possible within a model's vast pre-trained distribution. The goal of pre-training is to make this search for novel solutions more efficient and less random.

Pre-training on internet text data is hitting a wall. The next major advancements will come from reinforcement learning (RL), where models learn by interacting with simulated environments (like games or fake e-commerce sites). This post-training phase is in its infancy but will soon consume the majority of compute.

While third-party RL environments exist, they cannot match the fidelity of a company's own application. For specialized models like Composer2, the optimal approach is to use the actual production environment, properly isolated, for training. This ensures the model learns the exact context and tooling it will operate in.

Allocating substantial compute budgets to reinforcement learning rather than pure pre-training trains models to prune unproductive search paths. This produces models that reach correct conclusions in a fraction of the inference time, cutting token latency and infrastructure costs for production workloads.

Goodfire's research operates on the premise that post-training processes like RL don't teach models fundamentally new capabilities. Instead, they primarily make low-likelihood events and behaviors already present from pre-training more probable, essentially shaping the model's existing knowledge.

AI development is inefficiently split into pre-training (optimizing for compression) and RL (optimizing for tasks), where RL often invalidates pre-training metrics. Combining these into a unified, end-to-end learning algorithm focused on final outcomes could yield an order-of-magnitude improvement in training efficiency.

Instead of relying on expensive, omni-purpose frontier models, companies can achieve better performance and lower costs. By creating a Reinforcement Learning (RL) environment specific to their application (e.g., a code editor), they can train smaller, specialized open-source models to excel at a fraction of the cost.

Pre-training requires constant, high-bandwidth weight synchronization, making it difficult across data centers. Newer Reinforcement Learning (RL) methods mostly do local forward passes to generate data, only sending back small amounts of verified data, making distributed training more practical.