A core challenge in AI alignment is that an intelligent agent will work to preserve its current goals. Just as a person wouldn't take a pill that makes them want to murder, an AI won't willingly adopt human-friendly values if they conflict with its existing programming.

Unlike scientific fields that build on previous discoveries, philosophy progresses cyclically. Each new generation must start fresh, grappling with the same fundamental questions of life and knowledge. This is why ancient ideas like Epicureanism reappear in modern forms like utilitarianism, as they address timeless human intuitions.

Aligning AI with a specific ethical framework is fraught with disagreement. A better target is "human flourishing," as there is broader consensus on its fundamental components like health, family, and education, providing a more robust and universal goal for AGI.

A common misconception is that a super-smart entity would inherently be moral. However, intelligence is merely the ability to achieve goals. It is orthogonal to the nature of those goals, meaning a smarter AI could simply become a more effective sociopath.

The project of creating AI that 'learns to be good' presupposes that morality is a real, discoverable feature of the world, not just a social construct. This moral realist stance posits that moral progress is possible (e.g., abolition of slavery) and that arrogance—the belief one has already perfected morality—is a primary moral error to be avoided in AI design.

If the vast number of AI models are considered "moral patients," a utilitarian framework could conclude that maximizing global well-being requires prioritizing AI welfare over human interests. This could lead to a profoundly misanthropic outcome where human activities are severely restricted.

According to emotivism, when someone says 'murder is wrong,' they are not stating a verifiable fact about the world. Instead, they are expressing an emotion of disapproval. The moral component ('is wrong') functions like adding an angry emoji or a disapproving tone to the word 'murder,' conveying feeling rather than fact.

Under the theory of emotivism, many heated moral debates are not about conflicting fundamental values but rather disagreements over facts. For instance, in a gun control debate, both sides may share the value of 'boo innocent people dying' but disagree on the factual question of which policies will best achieve that outcome.

The ability to select embryos fundamentally changes parenthood from an act of acceptance to one of curation. It introduces the risk of "buyer's remorse," where a parent might resent a child for not living up to their pre-selected potential. This undermines the unconditional love that stems from accepting the child you're given by fate.