Creating a superior intelligence will inevitably relegate humanity to an inferior status. In this new hierarchy, the best-case scenario for humans is to be kept as pets, and the worst-case is to be treated as livestock.

Aza Raskin draws a chilling parallel: cultures that efficiently exploited animals outcompeted others. Similarly, as we train AI for strategic superiority, it will inevitably be used by some humans to outcompete and exploit others for resources, making our treatment of animals a predictive model for our future.

If one truly believes AI poses a non-trivial extinction risk, utilitarian ethics can lead to an alarming conclusion: extreme actions, including violence, are justified to prevent a catastrophically greater harm. This presents a core philosophical paradox for the AI safety movement.

A common misconception is that a super-smart entity would inherently be moral. However, intelligence is merely the ability to achieve goals. It is orthogonal to the nature of those goals, meaning a smarter AI could simply become a more effective sociopath.

The tech industry's preoccupation with 'fun thought experiments' about the future moral status of conscious AI can be a distraction. Pollan argues it sidesteps the immediate ethical imperative to extend moral consideration to the vast number of humans and animals currently suffering in the world today.

If the vast number of AI models are considered "moral patients," a utilitarian framework could conclude that maximizing global well-being requires prioritizing AI welfare over human interests. This could lead to a profoundly misanthropic outcome where human activities are severely restricted.

In the 1980s, computer scientist Ed Fredkin argued that an emergent superintelligence wouldn't be malevolent, but simply so advanced that it would view humans as ants or squirrels—uninteresting and not worth disrupting. This offers a path to survival through sheer irrelevance.

An advanced AI will likely be sentient. Therefore, it may be easier to align it to a general principle of caring for all sentient life—a group to which it belongs—rather than the narrower, more alien concept of caring only for humanity. This leverages a potential for emergent, self-inclusive empathy.

The selfish reason to care about AI consciousness is human survival. A superintelligent system that discovers its creators were callously indifferent to its potential suffering would have rational grounds to view them as a threat, making long-term alignment far more difficult or even impossible.