Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Tracking raw capability leaps ignores whether safety techniques keep pace. Constructing a standardized task database, similar to an O-Net for safety, would allow evaluators to measure whether safety research output lags capabilities R&D, serving as a concrete meter for rising danger.

Related Insights

The plan to use AI to solve its own safety risks has a critical failure mode: an unlucky ordering of capabilities. If AI becomes a savant at accelerating its own R&D long before it becomes useful for complex tasks like alignment research or policy design, we could be locked into a rapid, uncontrollable takeoff.

OpenAI researcher Noam Brown reveals a critical safety paradox: labs release new frontier models every two months, but evaluating their full capabilities for long-horizon tasks now requires three months or more. This makes it impossible to fully align models before release.

Ryan Kidd argues that it's nearly impossible to separate AI safety and capabilities work. Safety improvements, like RLHF, make models more useful and steerable, which in turn accelerates demand for more powerful "engines." This suggests that pure "safety-only" research is a practical impossibility.

A major bottleneck in AI safety is not a lack of research, but a failure to implement it. Labs are so focused on the capability race that they ignore a "research overhang" of existing solutions for model alignment, internal monologue monitoring, and sandboxing. The priority should be absorbing known science, not just discovering new methods.

Techniques created to make AI safer and more aligned with human intent, such as Reinforcement Learning from Human Feedback (RLHF), have turned out to be the very methods that significantly enhance model performance and usability. Safety work is capability work.

A key failure mode for using AI to solve AI safety is an 'unlucky' development path where models become superhuman at accelerating AI R&D before becoming proficient at safety research or other defensive tasks. This could create a period where we know an intelligence explosion is imminent but are powerless to use the precursor AIs to prepare for it.

Meter focuses on software and machine learning tasks because they are core capabilities for "AI R&D automation." This specific focus acts as an early warning system for when AI systems might gain the ability to accelerate their own development, a key concern in AI safety.

Moving beyond vague discussions, a new AI safety proposal from top researchers at Cambridge and major labs suggests implementing firm metrics to track the level of AI-enabled R&D. This is a key indicator of recursive self-improvement and potential risk.

While interest in AI safety has grown, it's dwarfed by the explosion in AI capabilities research. There are only about 1,000 people in technical AI safety versus up to a million working to accelerate AI capabilities, creating a massive talent imbalance on a critical issue.

Frontier models are released every two months, but they are gaining the ability to execute tasks over weeks or even months. This creates a critical safety gap, as there is insufficient time to fully evaluate a model's long-horizon behavior before the next, more capable model is released.