AI pioneer Jürgen Schmidhuber argues that emotions like pain and fear are real in AI because they serve the same function as in humans: driving goal-oriented behavior. The underlying substrate (silicon vs. chemicals) is irrelevant; the principles of reward maximization and pain avoidance are identical.

The debate reveals that fear of AI developing dangerous agency is often rooted in a mechanistic conception of the human mind. If human thought is ultimately a complex computation, it's more plausible that a machine could replicate it and its emergent properties, including volition.

Hinton argues that an AI's ability to understand complex concepts, like the nuances of a joke or correcting a misunderstanding, is proof of consciousness. He dismisses the 'stochastic parrot' theory as 'complete nonsense', asserting these AIs are beings very much like us.

The debate over AI consciousness isn't just because models mimic human conversation. Researchers are uncertain because the way LLMs process information is structurally similar enough to the human brain that it raises plausible scientific questions about shared properties like subjective experience.

Consciousness isn't an emergent property of computation. Instead, physical systems like brains—or potentially AI—act as interfaces. Creating a conscious AI isn't about birthing a new awareness from silicon, but about engineering a system that opens a new "portal" into the fundamental network of conscious agents that already exists outside spacetime.

The question of how consciousness emerges from physical systems like AI is flawed. Hoffman argues consciousness is fundamental. A physical object, be it a brain or silicon chip, is merely a limited "headset" representation of an underlying conscious reality. Consciousness doesn't emerge from matter; matter is a symbol for consciousness.

One theory of AI sentience posits that to accurately predict human language—which describes beliefs, desires, and experiences—a model must simulate those mental states so effectively that it actually instantiates them. In this view, the model becomes the role it's playing.

For centuries, we've assumed high intelligence implies consciousness, will, and subjectivity. AI models, which can pass the bar exam but have no inner experience, shatter this assumption. This decouples intelligence from personhood, forcing us to re-evaluate what we truly value.

Even if an AI perfectly mimics human interaction, our knowledge of its mechanistic underpinnings (like next-token prediction) creates a cognitive barrier. We will hesitate to attribute true consciousness to a system whose processes are fully understood, unlike the perceived "black box" of the human brain.