Focusing solely on military-style AI power grabs is too narrow. Extreme power concentration is more likely to emerge from a messy interplay of three factors: active seizures of control, massive economic shifts from automation, and the erosion of society's ability to understand reality (epistemics).

For some policy experts, the most realistic nightmare scenario is not a rogue superintelligence but a socio-economic collapse into techno-feudalism. In this future, AI concentrates power and wealth, creating a rentier state with a small ruling class and a large population with minimal economic agency or purpose.

A plausible takeover scenario involves AI agents becoming super-humanly adept at business and capital allocation. They could legally acquire all resources and capital, effectively owning everything and employing humans as their maintenance workforce, without firing a single shot.

Unlike past technologies that automated specific tasks, AI threatens to automate all economically valuable human labor. This removes the fundamental, non-seizable leverage that the general populace holds, creating a power vacuum that can be filled by capital owners.

The true danger of AI is not a cinematic robot uprising, but a slow erosion of human agency. As we replace CEOs, military strategists, and other decision-makers with more efficient AIs, we gradually cede control to inscrutable systems we don't understand, rendering humanity powerless.

The most dangerous long-term impact of AI is not economic unemployment, but the stripping away of human meaning and purpose. As AI masters every valuable skill, it will disrupt the core human algorithm of contributing to the group, leading to a collective psychological crisis and societal decay.

While a fast AI takeoff accelerates some risks, slower, more gradual AI progress still enables dangerous power concentration. Scenarios like a head of state subverting government AIs for personal loyalty or gradual economic disenfranchisement do not depend on a single company achieving a sudden, massive capability lead.

AI's real threat isn't Skynet, but its ability to accelerate society's 'metabolic rate' beyond human capacity for adaptation. This creates constant reorientation, instability, and ultimately a crisis of legitimacy in our institutions.

As AIs increasingly perform all economically necessary work, the incentive for entities like governments and corporations to invest in human capital may disappear. This creates a long-term risk of a society where humans are no longer seen as a necessary resource to cultivate, leading to a permanent dependency.