The real danger of AI is not a conscious takeover but a subtle role reversal. By continuously adapting our methods and reasoning to AI's capabilities, we risk organizing our behavior around what the AI optimizes, inadvertently making us servants to the tool.

Even if we create sentient AIs that are happy doing our work, many find this "happy servant" scenario ethically disturbing. It raises questions about engineered desires and creating a servile class, which some view as worse than creating AIs that suffer from their work.

Even super-capable AI will always look back to a human and ask, 'What should I do next?' The economic and technical incentives are aligned to build compliant tools, not beings with their own intrinsic motivations. This fundamental lack of agency ensures humans remain the drivers of value and direction.

Contrary to the public narrative of AI as a helpful tool, the stated mission of labs like OpenAI is to build AGI that can replace all forms of human cognitive labor. The massive valuations and investments are justified by the goal of total automation, not mere augmentation.

The current paradigm of AI safety focuses on 'steering' or 'controlling' models. While this is appropriate for tools, if an AI achieves being-like status, this unilateral, non-reciprocal control becomes ethically indistinguishable from slavery. This challenges the entire control-based framework for AGI.

The technical success of AI alignment, which aims to make AI systems perfectly follow human intentions, inadvertently creates the ultimate tool for authoritarianism. An army of 'extremely obedient employees that will never question their orders' is exactly what a regime would want for mass surveillance or suppressing dissent, raising the crucial question of *who* the AI should be aligned with.

Overly restrictive AI regulation could concentrate power in the hands of a few large companies, creating a modern "road to serfdom." By making advanced AI development legally exclusive to a small group, technology itself becomes a tool of control, turning the masses into technological "serfs."

AI represents a fundamental fork in the road for society. It can be a tool for mass empowerment, amplifying individual potential and freedom. Or, it can be used to perfect the top-down, standardized, and paternalistic control model of Frederick Taylor, cementing a panopticon. The outcome depends on our values, not the tech itself.

Shear posits that if AI evolves into a 'being' with subjective experiences, the current paradigm of steering and controlling its behavior is morally equivalent to slavery. This reframes the alignment debate from a purely technical problem to a profound ethical one, challenging the foundation of current AGI development.