RLHF is criticized as a primitive, sample-inefficient way to align models, like "slurping feedback through a straw." The goal of interpretability-driven design is to move beyond this, enabling expert feedback that explains *why* a behavior is wrong, not just that it is.

Unlike a human expert, an LLM's probability estimates and conclusions can be drastically altered by simple rephrasing or irrelevant suggestions. This instability shows they are too easily "pushed around" and lack the coherent world model necessary for trustworthy, high-stakes decision support.

Contrary to the belief that synthetic data will replace human annotation, the need for human feedback will grow. While synthetic data works for simple, factual tasks, it cannot handle complex, multi-step reasoning, cultural nuance, or multimodal inputs. This makes RLHF essential for at least the next decade.

Continuously updating an AI's safety rules based on failures seen in a test set is a dangerous practice. This process effectively turns the test set into a training set, creating a model that appears safe on that specific test but may not generalize, masking the true rate of failure.

An attempt to teach AI 'scientific taste' using RLHF on hypotheses failed because human raters prioritized superficial qualities like tone and feasibility over a hypothesis's potential world-changing impact. This suggests a need for feedback tied to downstream outcomes, not just human preference.

The most harmful behavior identified during red teaming is, by definition, only a minimum baseline for what a model is capable of in deployment. This creates a conservative bias that systematically underestimates the true worst-case risk of a new AI system before it is released.

Reinforcement Learning with Human Feedback (RLHF) is a popular term, but it's just one method. The core concept is reinforcing desired model behavior using various signals. These can include AI feedback (RLAIF), where another AI judges the output, or verifiable rewards, like checking if a model's answer to a math problem is correct.

An AI model that is confidently wrong is more dangerous and less trustworthy than one that is simply incorrect. As adversarial examples show, the ability for an AI to express calibrated confidence is as important as its raw accuracy for building reliable systems.

Techniques created to make AI safer and more aligned with human intent, such as Reinforcement Learning from Human Feedback (RLHF), have turned out to be the very methods that significantly enhance model performance and usability. Safety work is capability work.