Generative AI can produce the "miraculous" insights needed for formal proofs, like finding an inductive invariant, which traditionally required a PhD. It achieves this by training on vast libraries of existing mathematical proofs and generalizing their underlying patterns, effectively automating the creative leap needed for verification.

Anthropic's work on reading a model's internal "thoughts" is more than a safety feature; it's a new frontier for performance. The ability to "train the thoughts, not just the words" gives developers a direct lever to improve a model's internal reasoning, fix failures, and enhance reliability, moving interpretability from theory to practice.

There's a critical distinction between a proof (which establishes truth) and an explanation (which provides understanding). Even when a complex mathematical problem is solved, there remains an 'unsolved expository problem' of making the solution comprehensible. This need for clarity and intuition will remain a crucial area for human or AI effort, even after theorems are proven.

Expert mathematicians adopt formal tools like Lean not primarily to catch errors, but to offload tedious, low-level deductions. This automation allows them to operate at a higher level of abstraction and focus their cognitive energy on creative intuition and problem-solving strategy.

As AIs automate theorem proving and even explanation, the role of human mathematicians will shift. Instead of being creators, they will act as curators, using their taste and social connection to guide others through the vast, AI-generated landscape of mathematical ideas. Their value will lie in providing motivation and a human-centric narrative.

As AI models are used for critical decisions in finance and law, black-box empirical testing will become insufficient. Mechanistic interpretability, which analyzes model weights to understand reasoning, is a bet that society and regulators will require explainable AI, making it a crucial future technology.

Unlike classic theories based on simple equations, large AI models represent a new kind of scientific object. Rather than being mere predictive tools, they could be a novel form of explanation that we must learn to manipulate through new operations like distillation and merging, much like Mathematica made massive equations workable.

For AI systems to be adopted in scientific labs, they must be interpretable. Researchers need to understand the 'why' behind an AI's experimental plan to validate and trust the process, making interpretability a more critical feature than raw predictive power.

Even when a model performs a task correctly, interpretability can reveal it learned a bizarre, "alien" heuristic that is functionally equivalent but not the generalizable, human-understood principle. This highlights the challenge of ensuring models truly "grok" concepts.