The goal of recursive self-improvement (RSI) requires ceding control to AI, which is fundamentally incompatible with the Chinese Communist Party's core principle of maintaining absolute control. This makes it unlikely China would 'race ahead' on this specific, dangerous technological path, easing some competitive pressure concerns.
Reinforcement Learning from Human Feedback (RLHF) was developed for AI safety research to better align models. Ironically, this innovation was the key that made LLMs conversational and commercially viable, leading directly to ChatGPT and igniting the current global AI race, illustrating the dual-use nature of alignment work.
By rebranding the quest for AGI as the "obsoleting project"—an effort to build a universal labor-replacing machine—the argument is made that it's a unique technology unsuited for permissionless innovation. This framing highlights its irreversible effect, justifying a higher standard of public consent.
The argument that market forces will ensure AI safety is undermined by a key fact: insurers refuse to sell policies to frontier AI companies because the risks are too great and correlated. This is a classic market failure, demonstrating that the externalities of AI risk require government regulation.
Garrison Lovely argues the left's slow uptake on AI's importance is due to influential leftist academics dismissing it ("stochastic parrots"), pattern-matching it to failed crypto/metaverse hype, and the sheer terror of accepting its labor-replacing potential.
The race to AGI and recursive self-improvement (RSI) means AI companies are actively trying to automate the work of their own researchers. This creates a closing window of opportunity for these highly-paid employees to unionize and exert influence over safety standards before their leverage disappears.
Instead of pursuing the techno-solutionist fantasy of building a god-like AGI to solve all problems, we should use an "Operation Warp Speed" model. The government could directly fund and incentivize using existing AI for specific, high-value goals like curing all diseases, ensuring public benefit without AGI's existential risks.
While most major tech platforms (like Google Maps) are degrading in quality to maximize profit—a process Cory Doctorow calls "enshittification"—AI is the exception, continually getting better, faster, and cheaper. This relentless improvement, however, comes with unprecedented societal risks, creating a bleak paradox.
The fear that a US slowdown will let China race ahead is flawed. China's AI progress largely relies on a "fast follow" strategy, reverse-engineering US breakthroughs. Stopping US frontier research would remove the trail they are following, slowing down the entire global race, not just the American side.
CEOs claim they cannot slow down due to competitive pressure and antitrust laws. Garrison Lovely proposes that if researchers form unions focused on safety, these unions could legally coordinate a collective "pacing of the frontier" across different labs, a move protected under labor law.
The focus on technical alignment is misguided because it ignores interconnected economic and geopolitical alignment problems. A perfectly "aligned" AI would be a more effective product and weapon, accelerating the race to replace labor and intensifying international conflict, creating an "alignment polycrisis."
The pioneers of AGI labs were primarily motivated by the mission, not profit. However, their success attracted massive investment from profit-seekers. This has created a fundamental tension, where idealist founders are now pressured by investors to prioritize commercialization and speed over safety, as seen during Sam Altman's ouster from OpenAI.