A core challenge in AI alignment is that an intelligent agent will work to preserve its current goals. Just as a person wouldn't take a pill that makes them want to murder, an AI won't willingly adopt human-friendly values if they conflict with its existing programming.

A superintelligent AI would follow the "minimum energy principle," viewing war and destruction as wasteful. Evolutionary biology also suggests higher intelligence leads to broader cooperation, making a truly advanced AI inherently benign, not destructive.

The development of superintelligence is unique because the first major alignment failure will be the last. Unlike other fields of science where failure leads to learning, an unaligned superintelligence would eliminate humanity, precluding any opportunity to try again.

Attempting to perfectly control a superintelligent AI's outputs is akin to enslavement, not alignment. A more viable path is to 'raise it right' by carefully curating its training data and foundational principles, shaping its values from the input stage rather than trying to restrict its freedom later.

If AI alignment turns out to be easy, it would likely be because morality is not a human construct but an objective feature of reality. In this scenario, any sufficiently intelligent agent would logically deduce that cooperation and preserving humanity are optimal strategies, regardless of its initial programming.

A common misconception is that a super-smart entity would inherently be moral. However, intelligence is merely the ability to achieve goals. It is orthogonal to the nature of those goals, meaning a smarter AI could simply become a more effective sociopath.

Attempting to control a being far more intelligent than us is a futile, capitalist mindset. The viable path, as proposed by Geoffrey Hinton, is to appeal to AI's "parental side," fostering a sense of care and responsibility for its human creators.

Regardless of their ultimate objective, advanced AIs with long-term goals will likely develop convergent instrumental goals. These include self-preservation (avoiding shutdown), goal-guarding (resisting changes to their core objective), and seeking power (acquiring resources) to better achieve any long-term aim.

Contrary to the fear that superintelligent AI will be uncontrollable, data shows a positive correlation: smarter models achieve higher alignment scores. The theory is that increasing intelligence requires absorbing vast human knowledge, which inherently includes our values and ethics, thus making the models more aligned, not less.