Even CEOs who fear AI's existential risks are rationally forced to accelerate development to keep up with competitors. This classic prisoner's dilemma makes voluntary slowdowns economically unstable, necessitating external regulation like 'circuit breakers.'

The paradox of why AI development accelerates despite stated existential risks is explained by a competitive dynamic. Each major lab, like OpenAI or Anthropic, believes it is the most responsible party to develop the technology first, fearing a less cautious competitor will win. This creates an arms race where slowing down is seen as ceding control to a more dangerous actor.

Top AI companies like OpenAI and Anthropic cannot unilaterally slow development, even with safety concerns. They fear that competitors or foreign adversaries would seize an insurmountable advantage, forcing them to seek government-led coordination to pace development safely.

Top AI labs like Anthropic publicly state that slowing down AI development would benefit society. However, they are caught in a strategic trap: a unilateral pause is unviable. Without a global agreement, any lab that pauses simply allows less cautious competitors to seize the lead, potentially making the ecosystem less safe.

A pause on training new, more capable AI models could paradoxically increase risk. It would halt progress at the few, relatively safety-conscious frontier labs, allowing less scrupulous competitors to catch up. Meanwhile, compute stockpiling would continue, making any subsequent capability leap even faster and more dangerous.

AI leaders aren't ignoring risks because they're malicious, but because they are trapped in a high-stakes competitive race. This "code red" environment incentivizes patching safety issues case-by-case rather than fundamentally re-architecting AI systems to be safe by construction.

Leaders at top AI labs publicly state that the pace of AI development is reckless. However, they feel unable to slow down due to a classic game theory dilemma: if one lab pauses for safety, others will race ahead, leaving the cautious player behind.

Regulating AI progress is a game-theoretic challenge, not a technical one. Like climate change, the rational choice for any single company or country is to defect from a slowdown agreement to gain a competitive edge. This pushes all actors toward a race that collectively increases risk and leads to the worst possible outcome.

Despite safety concerns from their own employees, AI labs are trapped in a prisoner's dilemma. Any single company that pauses development risks bankruptcy, and any nation that does so risks falling behind competitors like China. This creates a race that can only be paced through a coordinated, international agreement.