Martin Wolf frames AI not as just a technology but as a philosophical pact. We are gaining a powerful servant that raises existential questions about humanity's purpose and creates terrifying risks like unaccountable decision-making and AI-run armies.

The existential threat from AI isn't about controlling the technology, but about humanity controlling itself. The challenge is a 'God test' requiring a moral upgrade—overcoming our innate, self-serving cognitive biases to achieve the global cooperation needed to manage AI safely.

If humanity creates a godlike superintelligence, its nature—good or bad—will not be random. It will be a direct reflection of the collective human choices, values, and market forces that served as its evolutionary environment. We are consciously selecting the traits of our future "god," making its arrival humanity's ultimate test.

The current state of AI development parallels early human evolution. Just as the invention of language enabled a step-function change in human collaboration and intelligence, AI agents now require their own 'language'—a set of shared protocols—to move beyond individual tasks and unlock collective problem-solving.

We often think of "human nature" as fixed, but it's constantly redefined by our tools. Technologies like eyeglasses and literacy fundamentally changed our perception and cognition. AI is not an external force but the next step in this co-evolution, augmenting what it means to be human.

The title of Wright's book, "The God Test," is a metaphor for the profound moral and cooperative test AI poses to humanity. He argues that surviving the AI revolution requires our species to achieve a higher level of global cooperation and overcome internal conflicts.

Wright's core thesis is that AI's rapid advancement is not just "learning." It's a process akin to evolution that reverse-engineers fundamental human cognitive functions—like representing meaning—without needing explicit instruction, suggesting its potential is vast and unpredictable.

Human intelligence is shaped by limitations like a finite lifespan and small brain, forcing efficient learning from sparse data. AI lacks these constraints, learning from lifetimes of data with massive compute. This fundamental difference means AI will naturally evolve into a distinct, non-human form of intelligence unless we explicitly engineer human-like biases into it.

Drawing from the theory of Cultural Materialism, technological infrastructure dictates a society's values. For instance, yoking an ox changed views on animal sanctity. As AI makes human economic output obsolete, our societal value system may shift to see humans as inefficient or even parasitic.