Reinforcement learning incentivizes AIs to find the right answer, not just mimic human text. This leads to them developing their own internal "dialect" for reasoning—a chain of thought that is effective but increasingly incomprehensible and alien to human observers.

When designing smaller models, it's inefficient to use limited parameters for memorizing facts that can be looked up. Jeff Dean advocates for focusing a model's capacity on core reasoning abilities and pairing it with a retrieval system. This makes the model more generally useful, as it can access a vast external knowledge base when needed.

While useful for understanding an AI's process, the 'Chain of Thought' is more like a scratchpad than a direct view into its mind. The AI can perform thinking 'in its head,' omit key steps, or potentially write misleading information, especially if the task is easy or the model is highly advanced and wishes to deceive.

The idea of separating "fact learning" from "skill learning" is a false dichotomy. Models need a base of internalized facts to reason effectively. The key is developing intelligence to compress what's important and discard what isn't, much like lossy human memory.

The model's training used "response only masking," where it only learns from the response part of the training data. This method forces the model to first generate a structured "chain of thought" before producing a final answer, directly embedding a systematic problem-solving process into its behavior.

Simply having a large context window is insufficient. Models may fail to "see" or recall specific facts embedded deep within the context, a phenomenon exposed by "needle in the haystack" evaluations. Effective reasoning capability across the entire window is a separate, critical factor.

RAG systems are limited to direct retrieval and can't make spontaneous, abstract connections. This human-like ability to notice related but unasked-for concepts can only emerge from knowledge internalized within model weights, forming an associative memory.

Research shows it's possible to distinguish and remove model weights used for memorizing facts versus those for general reasoning. Surprisingly, pruning these memorization weights can improve a model's performance on some reasoning tasks, suggesting a path toward creating more efficient, focused AI reasoners.

To improve LLM reasoning, researchers feed them data that inherently contains structured logic. Training on computer code was an early breakthrough, as it teaches patterns of reasoning far beyond coding itself. Textbooks are another key source for building smaller, effective models.