Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Kevin Roose estimates about 50 people at frontier AI labs make the key calls on slowing, stopping, or coordinating AGI. He interviewed roughly 150 people for the book, but says the relevant decision cluster is tiny and the public has almost no voice in it.

Related Insights

Many top AI CEOs openly admit the extinction-level risks of their work, with some estimating a 25% chance. However, they feel powerless to stop the race. If a CEO paused for safety, investors would simply replace them with someone willing to push forward, creating a systemic trap where everyone sees the danger but no one can afford to hit the brakes.

Top AI lab leaders, including Demis Hassabis (Google DeepMind) and Dario Amodei (Anthropic), have publicly stated a desire to slow down AI development. They advocate for a collaborative, CERN-like model for AGI research but admit that intense, uncoordinated global competition currently makes such a pause impossible.

The leaders of top AI labs have signed statements acknowledging AI could cause human extinction. Yet, a safety report gives them failing grades on 'existential safety,' finding it jarring that these same leaders are actively building superintelligence without any articulated plan for how to maintain human control over the technology.

Top AI researchers currently wield significant influence, able to force policy reversals at labs like Anthropic because their talent is indispensable. However, this power is temporary. Once recursive self-improvement (RSI) becomes effective, the models themselves will drive progress, concentrating power solely with leadership and diminishing researchers' leverage.

Leaders at top AI labs publicly state that the pace of AI development is reckless. However, they feel unable to slow down due to a classic game theory dilemma: if one lab pauses for safety, others will race ahead, leaving the cautious player behind.

Top AI CEOs are driven by a mutual fear that if a competitor achieves AGI first, that person could become a dictator. This "race to be first" is less about commercial success and more about a paranoid, high-stakes power grab to prevent a rival from seizing ultimate control.

The fundamental roadmap and safety guardrails of artificial intelligence are currently controlled by roughly 50 to 100 people in San Francisco. Roose warns that an insular cohort cannot safely steward a transformative technology confronting existential and economic risks.

Many leaders at frontier AI labs perceive rapid AI progress as an inevitable technological force. This mindset shifts their focus from "if" or "should we" to "how do we participate," driving competitive dynamics and making strategic pauses difficult to implement.

Convincing the 20,000 global experts capable of building AGI to stop is nearly impossible. This effort acts as a perpetual demotivation machine that can't afford a single failure. The moment one bright person defects, the arms race continues, making long-term prevention futile.

Researchers from OpenAI and Meta publicly signed the "Pacing the Frontier" statement, using stark language like "deadly race" and "runaway nuclear chain reaction." Their urgent calls for a coordinated slowdown reveal deep-seated fears among the very people building the technology, indicating the risks are not just theoretical.

AGI's fate rests with roughly 50 people | RiffOn