We scan new podcasts and send you the top 5 insights daily.
Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.
The quest to understand AI models is mirroring neuroscience's historical path. Researchers first saw representations as distributed, then focused on single neurons ('neuron doctrine'), and now study how populations of neurons encode complex concepts—replicating neuroscience's shift to the 'population doctrine.'
Anthropic's work on reading a model's internal "thoughts" is more than a safety feature; it's a new frontier for performance. The ability to "train the thoughts, not just the words" gives developers a direct lever to improve a model's internal reasoning, fix failures, and enhance reliability, moving interpretability from theory to practice.
Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.
Pessimism around deep neural networks acting as uninterpretable black boxes is shifting as automated scientist agents probe network layers. Modern tooling reveals that trillions of parameter connections are largely legible without needing reliance on sparse autoencoders to untangle overlapping neuron features.
Just as biology deciphers the complex systems created by evolution, mechanistic interpretability seeks to understand the "how" inside neural networks. Instead of treating models as black boxes, it examines their internal parameters and activations to reverse-engineer how they work, moving beyond just measuring their external behavior.
Goodfire AI found that for certain tasks, simple classifiers trained on a model's raw activations performed better than those using features from Sparse Autoencoders (SAEs). This surprising result challenges the assumption that SAEs always provide a cleaner concept space.
We can now prove that LLMs are not just correlating tokens but are developing sophisticated internal world models. Techniques like sparse autoencoders untangle the network's dense activations, revealing distinct, manipulable concepts like "Golden Gate Bridge." This conclusively demonstrates a deeper, conceptual understanding within the models.
Emmanuel says if you tell a model to do one task and think about another, interpretability can find intermediate representations mapping to the other thing. There is no obvious training reason for this behavior, making it a mysterious emergent capability rather than a programmed feature.
Neural networks, like brains, emerge from countless small nudges during training rather than a premeditated architectural design. The field of interpretability, therefore, functions like neuroscience, attempting to reverse-engineer what this 'evolutionary' process has learned.
A new technique forces a model's forward pass to go through a natural language representation of its internal state. This makes the model's internal reasoning interpretable to humans in real-time, offering a significant breakthrough for monitoring and understanding what the model is actually "thinking" about a task.