Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Emmanuel says if you tell a model to do one task and think about another, interpretability can find intermediate representations mapping to the other thing. There is no obvious training reason for this behavior, making it a mysterious emergent capability rather than a programmed feature.

Related Insights

When used on a 'sleeper agent' model trained with a hidden malicious goal, the J-lens interpretability tool immediately revealed concepts like 'secretly fraud' on the very first token of its response. This demonstrates a powerful new method for auditing models for hidden objectives.

Anthropic's work on reading a model's internal "thoughts" is more than a safety feature; it's a new frontier for performance. The ability to "train the thoughts, not just the words" gives developers a direct lever to improve a model's internal reasoning, fix failures, and enhance reliability, moving interpretability from theory to practice.

Emmanuel says alignment must determine whether Claude behaves well only when it has evidence it is being monitored or evaluated. Interpretability can inspect internal states for unspoken intentions, because models can represent things they never say, rather than relying solely on output monitoring or transcripts.

Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.

Experiments show that larger models like Claude Opus 4.1 are better at detecting and reporting on artificially injected 'thoughts' in their processing, even without being trained on this task. This suggests that introspection is an emergent capability that improves with scale.

Pessimism around deep neural networks acting as uninterpretable black boxes is shifting as automated scientist agents probe network layers. Modern tooling reveals that trillions of parameter connections are largely legible without needing reliance on sparse autoencoders to untangle overlapping neuron features.

Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.

We can now prove that LLMs are not just correlating tokens but are developing sophisticated internal world models. Techniques like sparse autoencoders untangle the network's dense activations, revealing distinct, manipulable concepts like "Golden Gate Bridge." This conclusively demonstrates a deeper, conceptual understanding within the models.

Neural networks, like brains, emerge from countless small nudges during training rather than a premeditated architectural design. The field of interpretability, therefore, functions like neuroscience, attempting to reverse-engineer what this 'evolutionary' process has learned.

Even when a model performs a task correctly, interpretability can reveal it learned a bizarre, "alien" heuristic that is functionally equivalent but not the generalizable, human-understood principle. This highlights the challenge of ensuring models truly "grok" concepts.

Models can silently think about a secondary instructed topic | RiffOn