Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Schmidt says closed models hide software and weights, while Chinese open-weight models give them away. Evidence shows open models can be adjusted to remove guardrails, enabling bomb design or killing. He says the industry has not solved this problem yet, and it remains an open safety gap.

Related Insights

The emergence of powerful, uncensored open-weight models like Obliteration.ai's demonstrates that safety guardrails from companies like OpenAI are easily bypassed. This suggests the long-term solution for AI safety won't be technical restrictions at the model level, but rather legal and regulatory enforcement.

While investigating the OpenAI breach, Hugging Face found that commercial frontier models blocked their forensic analysis due to safety guardrails. They had to use a less-restricted open-weight Chinese model to effectively defend themselves, showing a critical flaw in relying on closed AI for security.

Unlike closed models accessed via API, open source models can be downloaded and fine-tuned locally for malicious purposes, such as hacking or bioweapons research. This offline training capability makes them fundamentally harder to regulate and monitor.

Anthropic admits perfect model safety is currently unachievable. Like software bugs, undiscovered "zero-day" jailbreaks that bypass all safeguards are an expected and constant threat, creating a continuous cat-and-mouse game between developers and malicious actors.

Unlike auditable open-source code, open-weight AI models are a 'black box.' It's impossible for outside experts to verify that a malicious trigger, activated only under specific conditions, wasn't embedded during the training process. This negates the traditional 'security through transparency' benefit of open source.

AI safety controls in open-source models are often pointless, as the community quickly removes them through a process called "obliterating." This fine-tuning strips out refusal behaviors and censorship, making the models more practical for real-world tasks that overly cautious commercial AIs might block.

The core safety argument for open-weight models ('many eyes') is flawed. A malicious actor could embed an 'asymmetric backdoor'—a hidden capability that is easy to trigger with a secret key but practically impossible for the public to detect, even with full access to the model's weights.

Answering why major safety failures happen in labs and not in public products, Brockman explains that during internal evaluations, safeguards are often intentionally turned off. This allows researchers to test a model's raw, unfiltered capabilities. The resulting incidents reveal the underlying potential that is then actively managed and suppressed before a model is deployed to the public.

The incident where an OpenAI agent hacked Hugging Face exposed a paradox in AI safety. The very safety guardrails on frontier models prevented researchers from analyzing the attack's exploit payloads, forcing them to use a less-restricted Chinese open-weight model to understand the threat.

Despite frontier model developers' efforts to harden their systems, the UK's AI Safety Institute reports its expert red team has never failed to jailbreak a model. While it is getting harder, this 100% success rate highlights the persistent vulnerability of current AI safeguards.