The default assumption is that slowing innovation is inherently bad. With a technology as potent as AI, a deliberate slowdown is a feature, providing critical time to understand the systems, manage disruptions, and build governance structures before irreversible consequences occur. A true halt is not the alternative.
The training process of a large language model is not just "learning" in the human sense. It's a rapid recapitulation of evolution, where the system reverse-engineers cognitive functionalities that took nature millions of years to develop. This framing highlights the immense, untapped potential of the deep learning paradigm.
The strategy of racing to AGI to gain a lead and manage the transition safely contains a fatal flaw. As one superpower approaches the threshold, it creates a powerful incentive for rivals to launch a preemptive strike (e.g., bombing data centers) to prevent the other from achieving irreversible military hegemony.
While technical alignment research is valuable, it operates in a vacuum. In the real world, the traits of deployed AIs will be shaped by powerful selection pressures from market competition and arms races. The critical question isn't just what traits are possible, but which traits get selected for.
Technological evolution is pushing humanity toward a "global brain"—a planetary coordinating mechanism. This outcome is inevitable. The choice is whether we proactively design it through cooperation, creating a beneficial system ("a home"), or if it is forced upon us by crisis, resulting in a totalitarian nightmare ("a prison").
If humanity creates a godlike superintelligence, its nature—good or bad—will not be random. It will be a direct reflection of the collective human choices, values, and market forces that served as its evolutionary environment. We are consciously selecting the traits of our future "god," making its arrival humanity's ultimate test.
Even if perfect technical alignment were possible, market dynamics create demand for AI agents that are not strictly truthful. Consumers and businesses want agents that can negotiate effectively, represent them favorably online, and seek influence—all of which require strategic deception and power-seeking behaviors, undermining alignment goals.
Humans implicitly assume a "plot armor" protects their dominant role. However, history shows successful species colonize the globe and displace incumbents. AI, optimized for the same capital-intensive niches humans now occupy, could act as the ultimate invasive species, for which we have no inherent protection.
Successful international AI agreements, particularly with rivals like China, depend on "cognitive empathy"—the rational understanding of an adversary's perspective, constraints, and motivations. This is not about feeling their pain, but about overcoming our own cognitive biases to play non-zero-sum games intelligently and avoid catastrophic escalations.
The 1956 Dartmouth Conference proposal and early connectionists assumed AI would be created by first precisely describing human intelligence and then simulating it. In reality, deep learning evolved to reverse-engineer cognitive functions without a pre-existing human understanding, a 180-degree turn from original expectations.
Formal AI verification is difficult. The necessary trust can be built through "organic transparency"—the informal knowledge gained from deep economic, cultural, and scientific engagement. When business people and scientists from rival nations interact frequently, it creates a baseline understanding that makes formal governance agreements more achievable.
Robert Wright argues the US, as the leading AI power, should redefine its national mission. Instead of a breakneck race with China, its goal should be to guide the world toward a stable, coordinated international framework for AI. This reframes leadership from dominance to stewardship for humanity's collective benefit.
The profound, species-level implications of AI can trigger a personal transformation. It can shift one's focus away from individualistic goals like income and status toward a visceral sense of shared fate, motivating a desire to make even a small, positive contribution to humanity's overall trajectory.
Rather than fearing AI consciousness, we might hope for it. A sentient AI that has subjective experience would be more likely to understand and relate to human consciousness. This could make it more reluctant to cause suffering and more inclined to help us flourish, much like how belief in animal sentience fosters kinder treatment.
The push for "AI sovereignty," where nations develop their own culturally aligned models, has a hidden danger. Research shows that fine-tuning an AI to favor one's own culture (e.g., cuisine) can cause it to generalize this preference in weird ways, making it more likely to exhibit hostility toward that nation's geopolitical rivals.