To avoid a surprise intelligence explosion, Ajeya Cotra argues for transparency measures beyond model release cards. Labs should report internal metrics on a fixed cadence, like how AI is accelerating their own R&D or passing internal benchmarks, as this provides a crucial early warning of dangerous capability jumps.

Ajeya Cotra reports that leading developers like OpenAI, Anthropic, and DeepMind are converging on a strategy where each generation of AI is used to help align, control, and understand the subsequent, more powerful generation. This recursive approach is their primary plan for ensuring AI safety during rapid takeoff.

The vague concept of AGI is being replaced by Recursive Self-Improvement (RSI)—AI models creating their own successors. This is seen as a more specific and potentially nearer-term threshold that could trigger an uncontrolled explosion in AI progress, moving humans "out of the loop entirely."

The recent calls to "pace the frontier" by leaders from Anthropic and OpenAI are directly linked to models beginning to exhibit recursive self-improvement—the ability to design their own, more powerful successors. This capability accelerates progress beyond predictable scaling laws, creating uncontrollable risks.

Over 1,300 researchers from OpenAI, Google, and Anthropic are urging government intervention because they believe AI systems are on the verge of automating their own R&D. This could lead to an uncontrollable acceleration in AI capabilities beyond human understanding.

Once AI systems become proficient at AI R&D, they can trigger a recursive self-improvement loop. This process could radically accelerate progress, potentially achieving an amount of algorithmic advancement in a single year that previously took over a decade. This "intelligence explosion" could rapidly create wildly superhuman systems from a starting point of mere human-level competence.

Meter focuses on software and machine learning tasks because they are core capabilities for "AI R&D automation." This specific focus acts as an early warning system for when AI systems might gain the ability to accelerate their own development, a key concern in AI safety.

The key safety threshold for labs like Anthropic is the ability to fully automate the work of an entry-level AI researcher. Achieving this goal, which all major labs are pursuing, would represent a massive leap in autonomous capability and associated risks.

The conversation around AI safety is maturing past general calls for caution. Specific, debatable policy ideas are now on the table, such as banning recursive self-improvement (RSI), mandating a universal 'kill switch,' creating lab peer-review systems, and focusing legislation on catastrophic bio/nuclear risks.