Jerry Tworek, a self-described "RL maximalist," found that scaling RL at OpenAI improved benchmarks but failed to solve real-world problems. The training data and evals were a closed loop, disconnected from the messy distribution of real user tasks, necessitating models that can learn at test time.

In domains like coding and math where correctness is automatically verifiable, AI can move beyond imitating humans (RLHF). Using pure reinforcement learning, or "experiential learning," models learn via self-play and can discover novel, superhuman strategies similar to AlphaGo's Move 37.

Unlike traditional software with deterministic outputs, generative AI systems require a new paradigm. Chip Huyen calls this "evaluation-driven development," where the focus shifts from writing fixed tests to building robust systems and guidelines for evaluating ambiguous, generative outputs.

To move beyond manual, "vibe-based" creation of AI skills, a quantifiable measurement system is needed. Trajectory RL is creating sandboxed benchmarks ("puzzle boxes") to objectively score skill performance, a necessary precursor to having AI agents write and improve skills themselves.

The frontier of AI training is moving beyond humans ranking model outputs (RLHF). Now, high-skilled experts create detailed success criteria (like rubrics or unit tests), which an AI then uses to provide feedback to the main model at scale, a process called RLAIF.

Reinforcement Learning with Human Feedback (RLHF) is a popular term, but it's just one method. The core concept is reinforcing desired model behavior using various signals. These can include AI feedback (RLAIF), where another AI judges the output, or verifiable rewards, like checking if a model's answer to a math problem is correct.

Focusing on which reinforcement learning algorithm is best (e.g., PPO vs. DPO) is misguided. The more critical factor is the quality and verifiability of the input data signal itself, which exists on a spectrum from subjective human preference (RLHF) to objective, verifiable truth.

The 'environment' concept extends beyond RL. It's a universal framework for any model interaction, encompassing the task, the harness, and the rubric. This same structure can be used for evaluations, A/B testing, prompt optimization, and synthetic data generation, making it a core building block for AI development.

As reinforcement learning (RL) techniques mature, the core challenge shifts from the algorithm to the problem definition. The competitive moat for AI companies will be their ability to create high-fidelity environments and benchmarks that accurately represent complex, real-world tasks, effectively teaching the AI what matters.