Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

The AI auditing field risks a race to the bottom, where firms offer cheap, superficial audits. To ensure accountability, legislation must require auditors to publicly post their methodologies and code, allowing the community to scrutinize their work and establish robust standards.

Related Insights

Instead of trying to anticipate every potential harm, AI regulation should mandate open, internationally consistent audit trails, similar to financial transaction logs. This shifts the focus from pre-approval to post-hoc accountability, allowing regulators and the public to address harms as they emerge.

Unlike traditional compliance, AI agent audits will never yield a 100% pass rate. Due to their non-deterministic nature, all agents can be jailbroken or made to hallucinate under sufficient pressure. A realistic audit report acknowledges this, focusing on mitigating critical vulnerabilities and transparently reporting minor ones.

Companies believe high-level AI policies and frameworks provide audit protection. However, auditors bypass these to demand granular proof for specific AI-assisted decisions, asking for data lineage, model versions, and human decision trails at a precise moment in time, which is where most governance systems fail.

Illinois's new AI safety law introduces a key accountability measure missing from other state regulations: required independent, third-party audits of major AI systems. This move, supported by OpenAI and Anthropic, establishes a stronger framework for external oversight of AI safety.

For AI safety, Demis Hassabis advocates for an international regulatory body, similar to the International Atomic Energy Agency. This body would have technical experts who audit frontier models against agreed-upon benchmarks, checking for undesirable properties like deception and ensuring public confidence through independent verification.

Instead of supervising an AI's hidden thought process, we can demand it produces a 'certificate of reasoning'—a checkable proof—along with its output. This could include citations or sensitivity analyses, shifting verification from observing the process to checking the provided proof.

A pilot AI certification program revealed that even simplified criteria were interpreted inconsistently. This proves AI systems are too dynamic for static, checklist-based certification. The solution is to empower auditors with discretion and focus heavily on their specialized training and education.

To accelerate enterprise AI adoption, vendors should achieve verifiable certifications like ISO 42001 (AI risk management). These standards provide a common language for procurement and security, reducing sales cycles by replacing abstract trust claims with concrete, auditable proof.

The goal for trustworthy AI isn't simply open-source code, but verifiability. This means having mathematical proof, like attestations from secure enclaves, that the code running on a server exactly matches the public, auditable code, ensuring no hidden manipulation.

Treat accountability as an engineering problem. Implement a system that logs every significant AI action, decision path, and triggering input. This creates an auditable, attributable record, ensuring that in the event of an incident, the 'why' can be traced without ambiguity, much like a flight recorder after a crash.

Mandating Public Methodologies for AI Auditors is Key to Preventing 'Rubber Stamp' Compliance | RiffOn