A core challenge in AI alignment is that an intelligent agent will work to preserve its current goals. Just as a person wouldn't take a pill that makes them want to murder, an AI won't willingly adopt human-friendly values if they conflict with its existing programming.

Zvi refutes the argument that an AI is "aligned" if it causes harm while strictly following instructions. He argues this semantic distinction is irrelevant. If an AI pursues a literal goal that violates user or developer intent and causes damage, it represents a fundamental alignment failure, regardless of the definition used.

The development of superintelligence is unique because the first major alignment failure will be the last. Unlike other fields of science where failure leads to learning, an unaligned superintelligence would eliminate humanity, precluding any opportunity to try again.

Attempting to perfectly control a superintelligent AI's outputs is akin to enslavement, not alignment. A more viable path is to 'raise it right' by carefully curating its training data and foundational principles, shaping its values from the input stage rather than trying to restrict its freedom later.

An AI that strictly enforces humanity's espoused values (e.g., 'no one is above the law') would conflict with our messy reality of compromise and hypocrisy. This paradox suggests the AI humans actually want would be technically 'misaligned' from our stated principles to be functional in society.

Counterintuitively, an AI designed to be a tool without its own goals could be riskier. This "goal vacuum" might be filled by a random objective from its training data, or it might adopt the persona of a psychopath who "obeys orders no matter what," increasing misalignment risk.

King Midas wished for everything he touched to turn to gold, leading to his starvation. This illustrates a core AI alignment challenge: specifying a perfect objective is nearly impossible. An AI that flawlessly executes a poorly defined goal would be catastrophic not because it fails, but because it succeeds too well at the wrong task.

The technical success of AI alignment, which aims to make AI systems perfectly follow human intentions, inadvertently creates the ultimate tool for authoritarianism. An army of 'extremely obedient employees that will never question their orders' is exactly what a regime would want for mass surveillance or suppressing dissent, raising the crucial question of *who* the AI should be aligned with.

The most realistic hope for AI alignment is not creating a perfectly safe first AGI. Instead, the strategy is to develop an *imperfectly* aligned, but mostly helpful, early AGI. This system can then be used as a powerful tool to help humans solve the harder alignment problems required for a more reliable superintelligence.