There's a significant gap between AI performance in simulated benchmarks and in the real world. Despite scoring highly on evaluations, AIs in real deployments make "silly mistakes that no human would ever dream of doing," suggesting that current benchmarks don't capture the messiness and unpredictability of reality.

Contrary to the belief that synthetic data will replace human annotation, the need for human feedback will grow. While synthetic data works for simple, factual tasks, it cannot handle complex, multi-step reasoning, cultural nuance, or multimodal inputs. This makes RLHF essential for at least the next decade.

Computer scientist Rich Sutton's "bitter lesson" is evolving. The new frontier for AI performance isn't just more pre-training data; it's vast amounts of "experiential data" from real-world user interactions. Models post-trained on this experience data are beginning to outperform those trained only on static, human-knowledge datasets.

A key risk in deploying AI is its inability to generalize to 'long-tail' or out-of-distribution events. Models trained on vast but finite data often fail when encountering novel situations common in the open-ended real world, such as a self-driving car mistaking a stop sign on a billboard for a real one.

Synthetic data serves as an efficient first step for training specialized AI, particularly when a larger model teaches a smaller one. However, it is insufficient on its own. The final, crucial stage always requires expensive "human signal"—feedback from subject matter experts—to achieve true performance.

Richard Sutton, author of "The Bitter Lesson," argues that today's LLMs are not truly "bitter lesson-pilled." Their reliance on finite, human-generated data introduces inherent biases and limitations, contrasting with systems that learn from scratch purely through computational scaling and environmental interaction.

Generalist CEO Pete Florence provides a tier list for robotics training data. He ranks "lived experience of the physical world" as S-tier, emphasizing the irreplaceable value of high-quality, real-world data. In contrast, he rates synthetic data from world models as F-tier, suggesting it is far less effective.

A common misconception is that simulation perfectly represents reality. In practice, it's a continuous loop: real-world data is required to tune simulator parameters, and this validation must be repeated until the gap between simulation and reality is small enough to trust the results.

Karpathy warns that training AIs on synthetically generated data is dangerous due to "model collapse." An AI's output, while seemingly reasonable case-by-case, occupies a tiny, low-entropy manifold of the possible solution space. Continual training on this collapsed distribution causes the model to become worse and less diverse over time.