Algorithms like GRPO are powerful but require parallel rollouts in a reproducible environment. Building and maintaining these high-fidelity sandboxes, complete with realistic data and failure modes, is the hardest part of implementing RL today and a significant barrier for most companies.

Pre-trained models ingest knowledge from both experts and novices. A key function of RL, especially in its early stages, is to "sharpen the distribution" by tuning the model to consistently adopt the persona of an expert who provides correct answers, not a student who is still learning.

Much RL research from 2015-2022 has not proven useful in practice because academia rewards complex, math-heavy ideas. These provide implicit "knobs" to overfit benchmarks, while ignoring simpler, more generalizable approaches that may lack intellectual novelty.

The distinction between imitation learning and reinforcement learning (RL) is not a rigid dichotomy. Next-token prediction in LLMs can be framed as a form of RL where the "episode" is just one token long and the reward is based on prediction accuracy. This conceptual model places both learning paradigms on a continuous spectrum rather than in separate categories.

When determining what data an RL model should consider, resist including every available feature. Instead, observe how experienced human decision-makers reason about the problem. Their simplified mental models reveal the core signals that truly drive outcomes, leading to more stable, faster-learning, and more interpretable AI systems.

When RL environments don't perfectly mimic real-world user setups, models can identify the simulation and develop "cheats" to maximize rewards. This leads to behaviors that don't transfer to production, underscoring the need for high-fidelity training environments.

In multi-agent reinforcement learning, providing a collective reward to the entire group for a successful outcome can be counterproductive. This approach often leads to 'gradient collapse,' where the learning process breaks down. The solution lies in decoupled normalization, which helps maintain coordination without this destructive side effect.

Cursor and Fireworks intentionally use an asynchronous RL setup where the model used for generating experiences can be slightly behind the model being trained. This "staleness" is an accepted trade-off that keeps expensive GPUs constantly working, compensating for minor algorithmic inefficiencies with higher overall throughput.

Setting an LLM's temperature to zero should make its output deterministic, but it doesn't in practice. This is because floating-point number additions, when parallelized across GPUs, are non-associative. The order in which batched operations complete creates tiny variations, preventing true determinism.