Reinforcement learning achieves superhuman results not by inventing alien concepts, but by surfacing and combining rare behaviors that are already possible within a model's vast pre-trained distribution. The goal of pre-training is to make this search for novel solutions more efficient and less random.

Pre-training on internet text data is hitting a wall. The next major advancements will come from reinforcement learning (RL), where models learn by interacting with simulated environments (like games or fake e-commerce sites). This post-training phase is in its infancy but will soon consume the majority of compute.

AI labs like Anthropic find that mid-tier models can be trained with reinforcement learning to outperform their largest, most expensive models in just a few months, accelerating the pace of capability improvements.

Dario Amodei views the distinction between RL and pre-training scaling as a red herring. He argues that, just like early language models needed broad internet-scale data to generalize (GPT-2 vs. GPT-1), RL needs to move beyond narrow tasks to a wide variety of environments to achieve true generalization.

The two greatest AI achievements are generative AI (mimicking human knowledge) and deep reinforcement learning (discovering superhuman strategies). The grand challenge, and the future of AI, is to fuse these two threads into a single system that can both leverage existing knowledge and innovate beyond it.

Beyond supervised fine-tuning (SFT) and human feedback (RLHF), reinforcement learning (RL) in simulated environments is the next evolution. These "playgrounds" teach models to handle messy, multi-step, real-world tasks where current models often fail catastrophically.

Pre-trained models ingest knowledge from both experts and novices. A key function of RL, especially in its early stages, is to "sharpen the distribution" by tuning the model to consistently adopt the persona of an expert who provides correct answers, not a student who is still learning.

Static data scraped from the web is becoming less central to AI training. The new frontier is "dynamic data," where models learn through trial-and-error in synthetic environments (like solving math problems), effectively creating their own training material via reinforcement learning.

The 'environment' concept extends beyond RL. It's a universal framework for any model interaction, encompassing the task, the harness, and the rubric. This same structure can be used for evaluations, A/B testing, prompt optimization, and synthetic data generation, making it a core building block for AI development.