Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

The intense U.S.-China rivalry prevents cooperation on AI safety, creating a dangerous vacuum. Without guardrails or arms control agreements, the risk of rogue AI agents or AI-powered weapons in nuclear or biological domains threatens to 'set back civilization.'

Related Insights

Robert Wright argues the most immediate existential danger isn't a superintelligent AI takeover, but the geopolitical competition. A breakneck race for AI supremacy could provoke one nation to launch a preemptive war to prevent the other from achieving a perceived decisive advantage.

The US and USSR, despite being adversaries, collaborated to prevent nuclear proliferation to rogue actors. A similar model can be applied to AI. The US and China share an interest in preventing powerful open-weight models from being used for cyber-attacks or bio-terrorism by third parties, creating a foundation for a safety dialogue.

In tabletop exercises simulating AI development, a common outcome is that nations race recklessly until a major incident (like a rogue superintelligence) forces a panicked global shutdown. This often leads to a complex three-way conflict between an international alliance, secret state projects, and the rogue AI.

The justification for accelerating AI development to beat China is logically flawed. It assumes the victor wields a controllable tool. In reality, both nations are racing to build the same uncontrollable AI, making the race itself, not the competitor, the primary existential threat.

The same governments pushing AI competition for a strategic edge may be forced into cooperation. As AI democratizes access to catastrophic weapons (CBRN), the national security risk will become so great that even rival superpowers will have a mutual incentive to create verifiable safety treaties.

A significant unintended consequence of US export controls is the creation of a climate of fear that prevents American AI safety organizations from collaborating with Chinese counterparts. This self-imposed barrier hinders global efforts to manage the risks of a technology that affects all of humanity.

The greatest risk in US-China tech competition is not one side getting ahead, but their technology stacks becoming mutually unintelligible. This lack of legibility creates a volatile environment ripe for catastrophic miscalculation and conflict, making a shared AI paradigm a strategic necessity for peace.

Despite intense technological competition, both the U.S. and China face a common threat from non-state actors like terrorist or criminal groups acquiring powerful AI models. This shared vulnerability presents a potential opportunity for cooperation on AI regulation and safeguards, even amid broader strategic rivalry.

Paradoxically, progress in AI alignment has made a global slowdown agreement impossible. Each side now trusts its own 'aligned' AI more than it trusts the other nation to uphold a deal. The perceived risk of being surpassed by a defector now outweighs the perceived risk of one's own AI going rogue, making a competitive race the rational choice.

Unlike the Cold War's 'mutually assured destruction,' if one nation secretly believes it can control a superintelligence, it will pursue it. The only path to a stable pause is if both the US and China agree that superintelligence is uncontrollable by anyone, leading to 'assured destruction' for all, thus aligning their incentives to truly halt development.