The plan to use AI to solve its own safety risks has a critical failure mode: an unlucky ordering of capabilities. If AI becomes a savant at accelerating its own R&D long before it becomes useful for complex tasks like alignment research or policy design, we could be locked into a rapid, uncontrollable takeoff.

The debate pitting AI safety against AI opportunity presents a false choice. Historical parallels, like the railroad industry, show that safety regulations (e.g., standardized tracks, air brakes) were essential for enabling greater speed, reliability, and economic potential. Trustworthy AI will unlock greater opportunity.

AI safety requires more than just technical controls. "Trust Engineering" is an emerging discipline that pairs human-centered design (e.g., clear visual signals from a self-driving car) with robust security infrastructure. This holistic approach manages user expectations and system behavior simultaneously.

OpenAI's health division serves a dual purpose: delivering societal benefits and providing a real-world, high-stakes environment for AI safety research. Problems like scalable oversight (supervising superhuman AI) move from theoretical exercises to practical necessities when models outperform physicians on narrow tasks, creating concrete feedback loops that accelerate safety progress.

Reinforcement Learning with Human Feedback (RLHF) is a popular term, but it's just one method. The core concept is reinforcing desired model behavior using various signals. These can include AI feedback (RLAIF), where another AI judges the output, or verifiable rewards, like checking if a model's answer to a math problem is correct.

Ryan Kidd argues that it's nearly impossible to separate AI safety and capabilities work. Safety improvements, like RLHF, make models more useful and steerable, which in turn accelerates demand for more powerful "engines." This suggests that pure "safety-only" research is a practical impossibility.

The view that safety measures hinder AI performance is a false dichotomy. A model's economic usefulness and profitability are directly tied to its controllability and predictability, making safety and alignment core product features rather than constraints.

As AI models become more powerful, they pose a dual challenge for human-centered design. On one hand, bigger models can cause bigger, more complex problems. On the other, their improved ability to understand natural language makes them easier and faster to steer. The key is to develop guardrails at the same pace as the model's power.

A key failure mode for using AI to solve AI safety is an 'unlucky' development path where models become superhuman at accelerating AI R&D before becoming proficient at safety research or other defensive tasks. This could create a period where we know an intelligence explosion is imminent but are powerless to use the precursor AIs to prepare for it.