Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

A cynical explanation for the race to build superintelligence is the immense power it would confer. The controller could develop technologies so advanced they would have a decisive advantage over all other global actors, akin to a group with guns facing one with swords.

Related Insights

The principle that governments must hold a monopoly on overwhelming force should extend to superintelligence. AI at that level has the power to disorient political systems and financial markets, making its private control untenable. The state cannot be secondary to any private entity in this domain.

The justification for accelerating AI development to beat China is logically flawed. It assumes the victor wields a controllable tool. In reality, both nations are racing to build the same uncontrollable AI, making the race itself, not the competitor, the primary existential threat.

The idea of nations collectively creating policies to slow AI development for safety is naive. Game theory dictates that the immense competitive advantage of achieving AGI first will drive nations and companies to race ahead, making any global regulatory agreement effectively unenforceable.

The narrative that AI could be catastrophic ('summoning the demon') is used strategically. It creates a sense of danger that justifies why a small, elite group must maintain tight control over the technology, thereby warding off both regulation and competition.

The first entity to achieve AGI could see it self-improve at an exponential rate, potentially achieving 20,000 years of progress overnight. This concept of "fast takeoff" makes any delay in the AI race, even for regulatory reasons, a potentially catastrophic strategic error.

The idea that AI development is a winner-take-all race to AGI is a compelling story that simplifies complex realities. This narrative is strategically useful as it creates a pretext for aggressive, 'do whatever it takes' behavior, sidestepping the messier nature of real-world conflict.

A strange dynamic exists where the tech leaders building AI are also the loudest voices warning of its potential to destroy humanity. This dual narrative of immense promise and existential threat serves to centralize their power, positioning them as the only ones who can both create and control this technology.

The immense strategic advantage offered by AI ensures its development will continue, regardless of safety concerns from insiders. Much like the Manhattan Project, which proceeded despite catastrophic risk, the logic of "if we don't, China will" makes unilateral cessation of research impossible for any major power.

Regardless of potential dangers, AI will be developed relentlessly. Game theory dictates that any nation or company that pauses or slows down will be at a catastrophic disadvantage to competitors who don't. This competitive pressure ensures the technology will advance without brakes.

The AI safety community fears losing control of AI. However, achieving perfect control of a superintelligence is equally dangerous. It grants godlike power to flawed, unwise humans. A perfectly obedient super-tool serving a fallible master is just as catastrophic as a rogue agent.

Controlling the First Superintelligence Could Grant a Decisive, World-Controlling Advantage | RiffOn