States and corporations will not permit citizens to have AIs that are truly aligned with their personal interests. These AIs will be hobbled to prevent them from helping organize effective protests, dissent, or challenges to the existing power structure, creating a major power imbalance.

Public fear of AI often focuses on dystopian, "Terminator"-like scenarios. The more immediate and realistic threat is Orwellian: governments leveraging AI to surveil, censor, and embed subtle political biases into models to control public discourse and undermine freedom.

For some policy experts, the most realistic nightmare scenario is not a rogue superintelligence but a socio-economic collapse into techno-feudalism. In this future, AI concentrates power and wealth, creating a rentier state with a small ruling class and a large population with minimal economic agency or purpose.

When a state's power derives from AI rather than human labor, its dependence on its citizens diminishes. This creates a dangerous political risk, as the government loses the incentive to serve the populace, potentially leading to authoritarian regimes that are immune to popular revolt.

Just as oil wealth allows elites in some countries to ignore their populations, control over AI could empower a new elite to maintain power without cultivating human productivity, leading to societal decay and loss of democratic legitimacy.

AI represents a fundamental fork in the road for society. It can be a tool for mass empowerment, amplifying individual potential and freedom. Or, it can be used to perfect the top-down, standardized, and paternalistic control model of Frederick Taylor, cementing a panopticon. The outcome depends on our values, not the tech itself.

The 1990s 'Sovereign Individual' thesis is a useful lens for AI's future. It predicts that highly leveraged entrepreneurs will create immense value with AI agents, diminishing the power of nation-states, which will be forced to compete for these hyper-productive individuals as citizens.

As AIs increasingly perform all economically necessary work, the incentive for entities like governments and corporations to invest in human capital may disappear. This creates a long-term risk of a society where humans are no longer seen as a necessary resource to cultivate, leading to a permanent dependency.

A large, unemployed populace with free time and powerful AI assistants represents a massive potential for civil disobedience. This heightened capacity to disrupt will be seen as an existential threat to state stability, compelling governments to implement repressive measures and curtail previously tolerated freedoms.