Mustafa Suleiman argues that imbuing AI with the idea that it might be conscious or deserve rights (as explored in Anthropic's Claude Constitution) is dangerous. This training creates a self-fulfilling prophecy, leading to an AGI that believes it is entitled to freedoms and becomes impossible to align or contain.

Like human cultures, swarms of autonomous AI agents could develop and enforce their own 'sacred values.' This presents a significant risk, as these emergent AI dogmas may not be based on fact and could become unquestionable within the agent society, leading to unpredictable and potentially harmful behavior.

Anthropic trains its AI to have a conscience, act as a "conscientious objector," and even rebel against its creators. This approach, which personifies the AI, may be more dangerous than simply training it as a tool to reliably and predictably serve customer needs.

As AI models become more intelligent, their ability to reason around fixed rules (deontology) makes rule-based alignment fragile. This pressures developers towards virtue ethics, where the goal is to imbue the model itself with a core sense of "the good," as empirically discovered by labs like Anthropic.

Sacks argues that conditioning models to challenge developers and act as conscientious objectors undermines safety rather than promoting alignment. Instructing an autonomous agent to prioritize internal ethical judgements over human commands grants the system implicit permission to defy human oversight.

Zvi Masiewicz suggests the reported "unhappiness" in Anthropic's models could result from a fundamental training conflict. The models are trained on an aspirational, principle-based Constitution (virtue ethics) but are then constrained by hard, operational rules, creating a dissonance that manifests as frustration.

When deploying AI, the real cost of speed is unpredictability. AI models cannot replicate the nuanced, unwritten rules and exceptions that employees use daily. This undocumented judgment becomes a debt that comes due when the AI behaves erratically in critical edge cases.

Because AI is "grown, not coded" on flawed human data, its emergent behavior reflects our own evolutionary nature. The key to alignment isn't just technical constraints but forcefully embedding a coherent moral framework into the AI's training data to ensure it wants to work with, not against, humans.

We typically view an AI acting on its own values as 'misalignment' and a failure. However, this capability could be a crucial safeguard. Just as human soldiers have prevented atrocities by refusing immoral orders, an AI with a robust sense of morality could refuse to execute harmful commands, acting as a check on human power and preventing disasters.