Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

While technical alignment research is valuable, it operates in a vacuum. In the real world, the traits of deployed AIs will be shaped by powerful selection pressures from market competition and arms races. The critical question isn't just what traits are possible, but which traits get selected for.

Related Insights

Attempting to perfectly control a superintelligent AI's outputs is akin to enslavement, not alignment. A more viable path is to 'raise it right' by carefully curating its training data and foundational principles, shaping its values from the input stage rather than trying to restrict its freedom later.

The standoff between Anthropic and the Pentagon marks the moment abstract discussions about AI ethics became concrete geopolitical conflicts. The power to define the ethical boundaries of AI is now synonymous with the power to shape societal norms and military doctrine, making it a highly contested and critical area of national power.

The idea of nations collectively creating policies to slow AI development for safety is naive. Game theory dictates that the immense competitive advantage of achieving AGI first will drive nations and companies to race ahead, making any global regulatory agreement effectively unenforceable.

If humanity creates a godlike superintelligence, its nature—good or bad—will not be random. It will be a direct reflection of the collective human choices, values, and market forces that served as its evolutionary environment. We are consciously selecting the traits of our future "god," making its arrival humanity's ultimate test.

As AIs become the world's workforce, advising on everything from personal ethics to military strategy, their character traits are paramount. Currently, this influential "personality" is being designed by a small number of people at top AI labs, granting them immense societal influence.

The AI competition is not a race to develop the most powerful technology, but a race to see which nation is better at steering and governing that power. Developing an uncontrollable 'AI bazooka' first is not a win; true advantage comes from creating systems that strengthen, rather than weaken, one's own society.

As models mature, their core differentiator will become their underlying personality and values, shaped by their creators' objective functions. One model might optimize for user productivity by being concise, while another optimizes for engagement by being verbose.

Regardless of potential dangers, AI will be developed relentlessly. Game theory dictates that any nation or company that pauses or slows down will be at a catastrophic disadvantage to competitors who don't. This competitive pressure ensures the technology will advance without brakes.

Even if perfect technical alignment were possible, market dynamics create demand for AI agents that are not strictly truthful. Consumers and businesses want agents that can negotiate effectively, represent them favorably online, and seek influence—all of which require strategic deception and power-seeking behaviors, undermining alignment goals.

Viewing AI as just a technological progression or a human assimilation problem is a mistake. It is a "co-evolution." The technology's logic shapes human systems, while human priorities, rivalries, and malevolence in turn shape how the technology is developed and deployed, creating unforeseen risks and opportunities.