For AI operating in the physical world, the goal isn't impossible perfection but perfect "explainability." Since systems will inevitably make mistakes, the ability to decompose an error, understand its root cause, and correct it is the most critical safety feature. Black-box outputs are unacceptable.

Mechanistic interpretability (Mekinterp) research has been slow due to its manual, ad-hoc nature. The guests argue that coding agents can automate the experimentation process, enabling large-scale, systematic analysis of AI models. The first science AI should automate is the science of understanding itself.

To trust an agentic AI, users need to see its work, just as a manager would with a new intern. Design patterns like "stream of thought" (showing the AI reasoning) or "planning mode" (presenting an action plan before executing) make the AI's logic legible and give users a chance to intervene, building crucial trust.

The ambition to fully reverse-engineer AI models into simple, understandable components is proving unrealistic as their internal workings are messy and complex. Its practical value is less about achieving guarantees and more about coarse-grained analysis, such as identifying when specific high-level capabilities are being used.

Just as biology deciphers the complex systems created by evolution, mechanistic interpretability seeks to understand the "how" inside neural networks. Instead of treating models as black boxes, it examines their internal parameters and activations to reverse-engineer how they work, moving beyond just measuring their external behavior.

As AI models are used for critical decisions in finance and law, black-box empirical testing will become insufficient. Mechanistic interpretability, which analyzes model weights to understand reasoning, is a bet that society and regulators will require explainable AI, making it a crucial future technology.

OpenAI identifies agent evaluation as a key challenge. While they can currently grade an entire task's trace, the real difficulty lies in evaluating and optimizing the individual steps within a long, complex agentic workflow. This is a work-in-progress area critical for building reliable, production-grade agents.

In traditional software, code is the source of truth. For AI agents, behavior is non-deterministic, driven by the black-box model. As a result, runtime traces—which show the agent's step-by-step context and decisions—become the essential artifact for debugging, testing, and collaboration, more so than the code itself.

A powerful evaluation technique is to ask an AI agent to analyze its own poor output. The agent can review its context and process, explain why it made a mistake, and even suggest how to update its own instructions to prevent future errors.