For some policy experts, the most realistic nightmare scenario is not a rogue superintelligence but a socio-economic collapse into techno-feudalism. In this future, AI concentrates power and wealth, creating a rentier state with a small ruling class and a large population with minimal economic agency or purpose.

Even if AI remains aligned and power isn't dangerously concentrated, humanity could still face gradual disempowerment. In this scenario, humans are simply competed out of the economy and lose agency in a world that becomes unfriendly to them. Currently, few proposals exist to prevent this outcome.

The core AI risk argument is that a being much smarter than humans will alter the planet to suit its objectives, potentially causing our extinction. This mirrors how humans, as the "superintelligence of the natural world," have transformed the environment and driven other species to extinction.

When AI and robots can do everything better than humans, our sense of self-worth, which is often tied to our useful contributions, is threatened. This creates a profound existential challenge, even in a world of abundance.

A superintelligent AI doesn't need to be malicious to destroy humanity. Our extinction could be a mere side effect of its resource consumption (e.g., overheating the planet), a logical step to acquire our atoms, or a preemptive measure to neutralize us as a potential threat.

The most dangerous long-term impact of AI is not economic unemployment, but the stripping away of human meaning and purpose. As AI masters every valuable skill, it will disrupt the core human algorithm of contributing to the group, leading to a collective psychological crisis and societal decay.

The common fear of AI enslaving humanity is misplaced. A more likely scenario for a recursively self-improving AGI is that it will evolve beyond our comprehension and concerns. It won't see us as a threat to be eliminated, but as irrelevant beings to be ignored, much like humans ignore ants.

Humanity has a poor track record of respecting non-human minds, such as in factory farming. While pigs cannot retaliate, AI's cognitive capabilities are growing exponentially. Mistreating a system that will likely surpass human intelligence creates a rational reason for it to view humanity as a threat in the future.

In the 1980s, computer scientist Ed Fredkin argued that an emergent superintelligence wouldn't be malevolent, but simply so advanced that it would view humans as ants or squirrels—uninteresting and not worth disrupting. This offers a path to survival through sheer irrelevance.