Focusing solely on military-style AI power grabs is too narrow. Extreme power concentration is more likely to emerge from a messy interplay of three factors: active seizures of control, massive economic shifts from automation, and the erosion of society's ability to understand reality (epistemics).

A CEO could embed undetectable loyalties to themselves into AI systems. If these systems are widely adopted by the government and military, the CEO could later trigger these loyalties to seize de facto control, bypassing traditional democratic and military chains of command without an overt conflict.

The principle that governments must hold a monopoly on overwhelming force should extend to superintelligence. AI at that level has the power to disorient political systems and financial markets, making its private control untenable. The state cannot be secondary to any private entity in this domain.

A strange dynamic exists where the tech leaders building AI are also the loudest voices warning of its potential to destroy humanity. This dual narrative of immense promise and existential threat serves to centralize their power, positioning them as the only ones who can both create and control this technology.

A cynical explanation for the race to build superintelligence is the immense power it would confer. The controller could develop technologies so advanced they would have a decisive advantage over all other global actors, akin to a group with guns facing one with swords.

Tom Davidson cautions that an AI development pause, if implemented badly, could concentrate immense power in an unaccountable government body. This group could abuse its authority to approve or deny AI projects, ironically enabling a state-led power grab.

As AI evolves into a significant source of power, private companies developing it cannot ignore governments. Ben Thompson argues that the state, defined by its monopoly on violence (the "people with guns"), will inevitably assert control over any technology this powerful, overriding corporate autonomy.

While AI alignment gets attention, the risk of AI concentrating immense power in the hands of a few actors (corporations or states) is arguably more neglected. This could enable unprecedented surveillance or create a single company with the economic power of a nation, posing a distinct and severe threat.

By constantly comparing AI's power to nuclear weapons, tech leaders are making a powerful argument against their own independence. If the technology is truly an existential threat, it logically follows that it should be government-controlled for national security, not managed by venture-backed startups.