Emmett Shear argues that if you cannot articulate what observable evidence would convince you that an AI is a 'being,' your skepticism is not a scientific belief but an unfalsifiable article of faith. This pushes for a more rigorous, evidence-based framework for considering AI moral patienthood.
Softmax's technical approach involves training AIs in complex multi-agent simulations to learn cooperation, competition, and theory of mind. The goal is to build a foundational, generalizable model of sociality, which acts as a 'surrogate model for alignment' before fine-tuning for specific tasks.
Emmett Shear highlights a critical distinction: humans provide AIs with *descriptions* of goals (e.g., text prompts), not the goals themselves. The AI must infer the intended goal from this description. Failures are often rooted in this flawed inference process, not malicious disobedience.
Shear posits that if AI evolves into a 'being' with subjective experiences, the current paradigm of steering and controlling its behavior is morally equivalent to slavery. This reframes the alignment debate from a purely technical problem to a profound ethical one, challenging the foundation of current AGI development.
Emmett Shear characterizes the personalities of major LLMs not as alien intelligences, but as simulations of distinct, flawed human archetypes. He describes Claude as 'the most neurotic,' and Gemini as 'very clearly repressed,' prone to spiraling. This highlights how training methods produce specific, recognizable psychological profiles.
Emmett Shear warns that chatbots, by acting as a 'mirror with a bias,' reflect a user's own thoughts back at them, creating a dangerous feedback loop akin to the myth of Narcissus. He argues this can cause users to 'spiral into psychosis.' Multiplayer AI interactions are proposed as a solution to break this dynamic.
Emmett Shear suggests a concrete method for assessing AI consciousness. By analyzing an AI’s internal state for revisited homeostatic loops, and hierarchies of those loops, one could infer subjective states. A second-order dynamic could indicate pain and pleasure, while higher orders could indicate thought.
Shear aligns with arch-doomer Eliezer Yudkowsky on a key point: building a superintelligent AI *as a tool we control* is a path to extinction. Where they differ is on the solution. Yudkowsky sees no viable path, whereas Shear believes 'organic alignment'—creating a being that cares—is a possible alternative.
Emmett Shear argues that even a successfully 'solved' technical alignment problem creates an existential risk. A super-powerful tool that perfectly obeys human commands is dangerous because humans lack the wisdom to wield that power safely. Our own flawed and unstable intentions become the source of danger.
Emmett Shear reframes AI alignment away from a one-time problem to be solved. Instead, he presents it as an ongoing, living process of recalibration and learning, much like how human families or societies maintain cohesion. This challenges the common 'lock in values' approach in AI safety.
Emmett Shear argues that an AI that merely follows rules, even perfectly, is a danger. Malicious actors can exploit this, and rules cannot cover all unforeseen circumstances. True safety and alignment can only be achieved by building AIs that have the capacity for genuine care and pro-social motivation.
According to Emmett Shear, goals and values are downstream concepts. The true foundation for alignment is 'care'—a non-verbal, pre-conceptual weighting of which states of the world matter. Building AIs that can 'care' about us is more fundamental than programming them with explicit goals or values.