Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

It seems counterintuitive to start new organizations if AGI is near. However, the alternative is inaction. The second-best time to start an AI safety company is today, as these entities are needed to build the solutions we'll rely on if we manage to slow down progress.

Related Insights

The belief that a future Artificial General Intelligence (AGI) will solve all problems acts as a rationalization for inaction. This "messiah" view is dangerous because the AI revolution is continuous and happening now. Deferring action sacrifices the opportunity to build crucial, immediate capabilities and expertise.

Framing AI governance as a short, time-limited 'pause' is a mistake. A better approach is a moratorium that ends not after a set time, but only when society's investment in safety and governance is 'commensurate' with the monumental risk of creating superintelligence, a standard we are currently failing to meet.

Instead of betting on specific AGI timelines, we should embrace uncertainty and adopt a 'broad timelines' strategy. This means the community should have a portfolio of projects: some focused on immediate, short-term interventions, and others on long-term capacity building like creating new institutions or academic fields.

The primary constraint to solving major AI safety challenges is the lack of experienced founders and leaders who can build and scale organizations. Ideas are plentiful, and funding follows great talent, making the 'founder bottleneck' the key problem to solve.

A near-future problem will be a surplus of philanthropic capital for AI safety but a deficit of high-quality organizations to absorb it. The founders who start organizations today are building the menu of investable options for the massive wave of funding to come.

Acknowledging their safety plans might be inadequate, leaders from multiple frontier labs have begun to seriously entertain a coordinated slowdown. This represents a major shift, as they also explore legal "safe harbors" to collaborate on safety without triggering antitrust violations, breaking the frame of the current race.

A fundamental tension within OpenAI's board was the catch-22 of safety. While some advocated for slowing down, others argued that being too cautious would allow a less scrupulous competitor to achieve AGI first, creating an even greater safety risk for humanity. This paradox fueled internal conflict and justified a rapid development pace.

The default assumption is that slowing innovation is inherently bad. With a technology as potent as AI, a deliberate slowdown is a feature, providing critical time to understand the systems, manage disruptions, and build governance structures before irreversible consequences occur. A true halt is not the alternative.

Even if the market would eventually build decision-making tools, their impact is time-sensitive. Waiting for commercial rollout might mean they arrive after AGI, too late to help navigate the riskiest period. Therefore, philanthropic or impact-driven acceleration, even by a few months, is highly valuable.

The risk from advanced AI is so imminent that the ideal time to slow down development has already passed. Debating the precise future moment for a pause is a dangerous distraction; action is needed now.