Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Emmanuel says models can compute an answer through intermediate steps, but they also have a separate gut-check that flags implausible results. For example, part of the model can react that four plus five equals 23 “can’t be right” without redoing the math, much like a human hunch.

Related Insights

Demis Hassabis likens current AI models to someone blurting out the first thought they have. To combat hallucinations, models must develop a capacity for 'thinking'—pausing to re-evaluate and check their intended output before delivering it. This reflective step is crucial for achieving true reasoning and reliability.

AI models frequently make confident errors when analyzing data or drawing inferences. Prompting tools to explicitly state when they do not know something and attach confidence scores quickly highlights low-confidence sections driven by missing data.

Probabilistic models excel at text because a sentence near the 'perfect' answer is usually still valid. In contrast, math is a 'hostile space' where the right answer is surrounded by wrong ones. A small deviation (e.g., 4.1 instead of 4) results in a completely incorrect output, explaining the challenge for LLMs.

Expert mathematicians don't just check proofs line-by-line; they assess the overall argument's structure and its broader implications. Current AI models fail at this crucial "big picture" validation. They can follow local logic but miss when a proof's fundamental approach is flawed or "too strong to be true."

AI models learn to reason like mathematicians—including backtracking and exploring dead ends—even though their training data (textbooks, papers) often presents clean, final proofs that hide the messy discovery process. This suggests the models are developing a general-purpose reasoning capability, not just mimicking final outputs.

Large Language Models learn the structure and language of mathematical solutions from vast text data. This allows them to generate convincing explanations and steps, but they don't perform actual calculations. Their "fluency" in math-like text is different from a calculator's logical execution, leading to confident but incorrect answers.

The model's training used "response only masking," where it only learns from the response part of the training data. This method forces the model to first generate a structured "chain of thought" before producing a final answer, directly embedding a systematic problem-solving process into its behavior.

The benefit of discrete reasoning (like generating tokens or tool calls) over a continuous 'neuralese' is error correction, analogous to why digital computing beat analog. A slightly wrong token can be 'rounded' to the correct one, preventing the compounding errors that would plague a purely continuous process.

Even when a model performs a task correctly, interpretability can reveal it learned a bizarre, "alien" heuristic that is functionally equivalent but not the generalizable, human-understood principle. This highlights the challenge of ensuring models truly "grok" concepts.

For tasks involving multi-step logic, evaluating only the final answer is insufficient. True correctness requires process-level evaluation, verifying each step in the AI's reasoning chain. A right conclusion reached through a faulty process is untrustworthy and indicates a model failure.

Models have a separate hunch mechanism that flags wrong answers | RiffOn