Reinforcement learning incentivizes AIs to find the right answer, not just mimic human text. This leads to them developing their own internal "dialect" for reasoning—a chain of thought that is effective but increasingly incomprehensible and alien to human observers.

The boom from LLMs was a 'shortcut' that mined intelligence from existing human data. This has limits. To achieve novel breakthroughs beyond that corpus, the field now re-integrates the original DeepMind philosophy of agents learning through interaction (like reinforcement learning) to generate truly new knowledge.

RL fine-tuning is less likely to cause catastrophic forgetting than SFT because it works within the model's existing pre-trained pathways, or "grooves." SFT, by contrast, makes much larger weight updates that can aggressively overwrite and destroy latent knowledge.

The argument that LLMs are just "stochastic parrots" is outdated. Current frontier models are trained via Reinforcement Learning, where the signal is not "did you predict the right token?" but "did you get the right answer?" This is based on complex, often qualitative criteria, pushing models beyond simple statistical correlation.

Pre-training on internet text data is hitting a wall. The next major advancements will come from reinforcement learning (RL), where models learn by interacting with simulated environments (like games or fake e-commerce sites). This post-training phase is in its infancy but will soon consume the majority of compute.

In domains like coding and math where correctness is automatically verifiable, AI can move beyond imitating humans (RLHF). Using pure reinforcement learning, or "experiential learning," models learn via self-play and can discover novel, superhuman strategies similar to AlphaGo's Move 37.

AI labs like Anthropic find that mid-tier models can be trained with reinforcement learning to outperform their largest, most expensive models in just a few months, accelerating the pace of capability improvements.

The two greatest AI achievements are generative AI (mimicking human knowledge) and deep reinforcement learning (discovering superhuman strategies). The grand challenge, and the future of AI, is to fuse these two threads into a single system that can both leverage existing knowledge and innovate beyond it.

Beyond supervised fine-tuning (SFT) and human feedback (RLHF), reinforcement learning (RL) in simulated environments is the next evolution. These "playgrounds" teach models to handle messy, multi-step, real-world tasks where current models often fail catastrophically.