The ultimate goal for companies like OpenAI and Anthropic is not just creating useful products like chatbots, but developing superintelligence—an AI that surpasses human cognitive ability in every domain, akin to the gap between a human and a mouse.
The motivation behind creating superintelligence is that it could apply its radical intelligence to solve humanity's biggest problems, like disease and scarcity. This could lead to a scale of abundance and flourishing currently unimaginable, echoing historical progress driven by technological advancements.
A cynical explanation for the race to build superintelligence is the immense power it would confer. The controller could develop technologies so advanced they would have a decisive advantage over all other global actors, akin to a group with guns facing one with swords.
Claims that AI CEOs use extinction risk as a marketing ploy are unconvincing. Many expressed these concerns long before leading major companies. Furthermore, highlighting catastrophic risk is a poor strategy for attracting investment and actively invites unwanted regulatory attention.
The dramatic slowdown of the nuclear power industry demonstrates that it is possible for governments to effectively halt the progress of a powerful technology. While this specific outcome may have been a net negative, it serves as a historical proof-of-concept for successfully implementing a global pause on AI development.
A significant barrier to voluntary safety pacts among AI companies is antitrust law. An agreement to slow development could be prosecuted as illegal anti-competitive collusion, as it would limit the technology available to consumers. This makes government-led frameworks essential for any coordinated industry action.
AIs develop internal models for complex concepts like human emotions "for free" simply by being trained to predict the next word in a vast text corpus. To accurately generate stories about anger, for example, the system must build a representation of anger, demonstrating emergent, general capabilities.
To prevent a reckless race, a proposed solution is a U.S.-China treaty to govern the resources needed for frontier AI. This would involve tracking and monitoring advanced AI chips in data centers and imposing a verifiable cap on the computational power used for any single training run.
The core safety challenge is that we have little understanding of how advanced AI systems function internally. We are essentially "growing" them through training, not engineering them with comprehensible parts. This means we cannot verify their true goals, making safety measures a gamble on observed behavior.
The global supply chain for cutting-edge AI chips is a major chokepoint, ideal for governance. Three companies design them, one (TSMC) manufactures over 90%, and one Dutch firm (ASML) makes the essential machinery. This concentration makes tracking and controlling compute resources feasible for a global coalition.
Political will on existential risk can change rapidly. President Reagan’s commitment to nuclear de-escalation was reportedly catalyzed by watching "The Day After," a TV movie depicting nuclear aftermath. This historical example suggests a similar visceral understanding of AI risk could spur leaders to action.
A pragmatic starting point for U.S.-China AI cooperation is to agree on verifiable red lines for proliferating dangerous dual-use capabilities, such as advanced cyberattack tools. This addresses a mutual security interest and builds the institutional trust and processes needed for more ambitious agreements on superintelligence.
CEOs from leading AI labs like Google DeepMind and Anthropic have publicly stated they would prefer to slow down development to address safety concerns. However, they feel compelled to continue the race because if they pause unilaterally, less cautious competitors, including state actors like China, will not.
Unlike human intelligence where skills like analytical reasoning and charisma are often decorrelated, AI systems can be trained to excel at a wide range of tasks simultaneously. General purpose learning algorithms can master both logical problems and persuasive communication, creating a more universally capable intelligence.