The 'use AI for safety' plan adopted by frontier labs is most likely to fail not because alignment techniques are ineffective, but because competitive pressures will prevent them from redirecting a meaningful fraction of their AI labor away from capabilities research and towards safety work when it matters most.

Top AI companies like OpenAI and Anthropic cannot unilaterally slow development, even with safety concerns. They fear that competitors or foreign adversaries would seize an insurmountable advantage, forcing them to seek government-led coordination to pace development safely.

The argument for rapidly advancing powerful AI is that only the leading labs can influence safety protocols. This 'stay in the lead to steer' philosophy creates a paradox: to mitigate AI risk, companies feel compelled to accelerate its development, potentially amplifying the very dangers they aim to control.

Top AI researchers currently wield significant influence, able to force policy reversals at labs like Anthropic because their talent is indispensable. However, this power is temporary. Once recursive self-improvement (RSI) becomes effective, the models themselves will drive progress, concentrating power solely with leadership and diminishing researchers' leverage.

Top AI policy experts are leaving government and academia for high-paying roles at frontier AI companies. This mirrors the earlier 'brain drain' of ML researchers and risks a future where AI regulation is overwhelmingly shaped by corporate-employed experts with vested interests.

Technical research is vital for governance because it provides concrete artifacts for policymakers. Demonstrations and evaluations showing dangerous AI behaviors make abstract risks tangible, giving policymakers a clear target for regulation, aligning with advice from figures like Jake Sullivan.

A key source of power for AI labs in government negotiations is the credible threat that their top researchers—a vital and mobile constituency—will revolt or quit if forced to comply with certain demands.

For any given failure mode, there is a point where further technical research stops being the primary solution. Risks become dominated by institutional or human factors, such as a company's deliberate choice not to prioritize safety. At this stage, policy and governance become more critical than algorithms.

While interest in AI safety has grown, it's dwarfed by the explosion in AI capabilities research. There are only about 1,000 people in technical AI safety versus up to a million working to accelerate AI capabilities, creating a massive talent imbalance on a critical issue.