Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Rob Wiblin argues capability improvements look less attractive near RSI because current models already help alignment and business, breakthroughs leak quickly, and the frontier is automated AI R&D itself. He says if your absence would let someone else do similar product work, you could likely do more neglected safety or governance work instead.

Related Insights

Rob Wiblin says technical staff have extraordinary bargaining power: they can slow work by not participating, leave for other impactful jobs, and speak more credibly than outsiders. But that influence ends once the models they build make them obsolete. Staff pressing employers on conscience also avoids the antitrust problems facing lab-leader agreements.

The 'use AI for safety' plan adopted by frontier labs is most likely to fail not because alignment techniques are ineffective, but because competitive pressures will prevent them from redirecting a meaningful fraction of their AI labor away from capabilities research and towards safety work when it matters most.

The plan to use AI to solve its own safety risks has a critical failure mode: an unlucky ordering of capabilities. If AI becomes a savant at accelerating its own R&D long before it becomes useful for complex tasks like alignment research or policy design, we could be locked into a rapid, uncontrollable takeoff.

A pause on training new, more capable AI models could paradoxically increase risk. It would halt progress at the few, relatively safety-conscious frontier labs, allowing less scrupulous competitors to catch up. Meanwhile, compute stockpiling would continue, making any subsequent capability leap even faster and more dangerous.

Top AI researchers currently wield significant influence, able to force policy reversals at labs like Anthropic because their talent is indispensable. However, this power is temporary. Once recursive self-improvement (RSI) becomes effective, the models themselves will drive progress, concentrating power solely with leadership and diminishing researchers' leverage.

While compute is a constraint on distribution, Greg Brockman argues that the actual bottleneck for developing more capable models is ensuring safety, security, and alignment. Progress on these fronts now dictates the pace at which the frontier can be advanced.

The race to AGI and recursive self-improvement (RSI) means AI companies are actively trying to automate the work of their own researchers. This creates a closing window of opportunity for these highly-paid employees to unionize and exert influence over safety standards before their leverage disappears.

A key failure mode for using AI to solve AI safety is an 'unlucky' development path where models become superhuman at accelerating AI R&D before becoming proficient at safety research or other defensive tasks. This could create a period where we know an intelligence explosion is imminent but are powerless to use the precursor AIs to prepare for it.

For any given failure mode, there is a point where further technical research stops being the primary solution. Risks become dominated by institutional or human factors, such as a company's deliberate choice not to prioritize safety. At this stage, policy and governance become more critical than algorithms.

The most likely reason AI companies will fail to implement their 'use AI for safety' plans is not that the technical problems are unsolvable. Rather, it's that intense competitive pressure will disincentivize them from redirecting significant compute resources away from capability acceleration toward safety, especially without robust, pre-agreed commitments.