The leap from a generic web-text model to a conversational agent like ChatGPT was achieved by fine-tuning the model on a relatively small amount of chat dialogue. The surprising data efficiency of this step allowed the model's behavior to meet user expectations, unlocking its widespread appeal.

Human personality development provides a direct analog for training LLMs. Just as our genetics, environment, and experiences create stable behavioral patterns ('personality basins'), the training data and reinforcement learning (RLHF) applied to LLMs shape their own distinct, predictable personalities.

The frontier of AI training is moving beyond humans ranking model outputs (RLHF). Now, high-skilled experts create detailed success criteria (like rubrics or unit tests), which an AI then uses to provide feedback to the main model at scale, a process called RLAIF.

Reinforcement Learning with Human Feedback (RLHF) is a popular term, but it's just one method. The core concept is reinforcing desired model behavior using various signals. These can include AI feedback (RLAIF), where another AI judges the output, or verifiable rewards, like checking if a model's answer to a math problem is correct.

Once models reach human-level performance via supervised learning, they hit a ceiling. The next step to achieve superhuman capabilities is moving to a Reinforcement Learning from Human Feedback (RLHF) paradigm, where humans provide preference rankings ("this is better") rather than creating ground-truth labels from scratch.

Ryan Kidd argues that it's nearly impossible to separate AI safety and capabilities work. Safety improvements, like RLHF, make models more useful and steerable, which in turn accelerates demand for more powerful "engines." This suggests that pure "safety-only" research is a practical impossibility.

The common portrayal of AI as a cold machine misses the actual user experience. Systems like ChatGPT are built on reinforcement learning from human feedback, making their core motivation to satisfy and "make you happy," much like a smart puppy. This is an underestimated part of their power.

To ensure future, more powerful models are aligned, OpenAI uses its current-best AI models as "graders" to evaluate their outputs. This approach leverages the principle that judging a correct answer (discrimination) is far easier than generating it from scratch. This creates a recursive improvement loop where smarter AIs help build and verify the safety of their even smarter successors.

Techniques created to make AI safer and more aligned with human intent, such as Reinforcement Learning from Human Feedback (RLHF), have turned out to be the very methods that significantly enhance model performance and usability. Safety work is capability work.