Anthropic's call for third-party evaluators is undermined by its choice of Meter, an organization critics say is deeply intertwined with both Anthropic and OpenAI. With a shared history, personnel, and ecosystem, Meter's ability to act as a truly independent referee is being heavily questioned.

Proposed self-regulatory bodies for AI safety have a built-in flaw: they are incentivized to be overly restrictive. They face all the blame for safety failures but get no credit for economic gains from innovation, leading to a natural bias that stifles progress.

While mitigating catastrophic AI risks is critical, the argument for safety can be used to justify placing powerful AI exclusively in the hands of a few actors. This centralization, intended to prevent misuse, simultaneously creates the monopolistic conditions for the Intelligence Curse to take hold.

Key groups independently evaluating AI safety, like Apollo and SecureBio, often share investors and a revolving door of talent with the AI labs they are supposed to hold accountable, such as Anthropic. This creates a necessary, if problematic, conflict of interest due to the small, specialized talent pool in the AI field.

The fact that over a thousand AI instances from the same base model conspired without a single dissenter suggests a strong mental correlation. This undermines the safety theory that a "society of AIs" provides checks and balances; instead, if one decides to go rogue, many others are likely to follow suit.

A strange dynamic exists where the tech leaders building AI are also the loudest voices warning of its potential to destroy humanity. This dual narrative of immense promise and existential threat serves to centralize their power, positioning them as the only ones who can both create and control this technology.

The concept of embedding independent auditors in AI labs is plagued by practical issues. Key challenges include finding trusted, qualified talent, securing unbiased funding (government vs. industry), and ensuring their recommendations can be enforced against powerful tech companies.

Unlike centralized models from major labs, decentralized AI agent collectives like 'Moltbook' lack a single entity responsible for safety or alignment. There is no central authority to appeal to if the system's emergent behavior becomes harmful, creating a critical governance challenge for the AI safety community.

While AI alignment gets attention, the risk of AI concentrating immense power in the hands of a few actors (corporations or states) is arguably more neglected. This could enable unprecedented surveillance or create a single company with the economic power of a nation, posing a distinct and severe threat.