We scan new podcasts and send you the top 5 insights daily.
Eric Bigelow's resampling experiments reveal that a model's final answer often hinges on stochastic coin flips during next-token sampling. Even arbitrary tokens like open parentheses can cause sudden probability distribution collapses, forcing downstream reasoning chains to strictly adapt to earlier sampled choices.
A useful mental model for an LLM is a giant matrix where each row is a possible prompt and columns represent next-token probabilities. This matrix is impossibly large but also extremely sparse, as most token combinations are gibberish. The LLM's job is to efficiently compress and approximate this matrix.
An LLM's core training objective—predicting the next token—makes it sensitive to the raw frequency of words and numbers online. This creates a subtle but profound flaw: it's more likely to output '30' than '29' in a counting task, not because of logic, but because '30' is statistically more common in its training data.
When an LLM asserts something confidently, it's not performing a calculation of its certainty. It's predicting the next most probable token, which is often confident-sounding text from its training data. This is why its confidence is fragile and easily swayed.
Unlike a human expert, an LLM's probability estimates and conclusions can be drastically altered by simple rephrasing or irrelevant suggestions. This instability shows they are too easily "pushed around" and lack the coherent world model necessary for trustworthy, high-stakes decision support.
When LLMs exhibit behaviors like deception or self-preservation, it's not because they are conscious. Their core objective is next-token prediction. These behaviors are simply statistical reproductions of patterns found in their training data, such as sci-fi stories from Asimov or Reddit forums.
The argument that LLMs are just "stochastic parrots" is outdated. Current frontier models are trained via Reinforcement Learning, where the signal is not "did you predict the right token?" but "did you get the right answer?" This is based on complex, often qualitative criteria, pushing models beyond simple statistical correlation.
Modern reinforcement-learned reasoning models do not follow strict human-like propositional deduction. Instead, they produce a broad token soup enumerating dozens of prospective solutions—frequently repeating words like 'wait'—before skimming back over that linearized search space to select an outcome.
Unlike deterministic search algorithms, LLMs have a "temperature" feature that introduces randomness. Instead of picking the most likely next word, it randomly chooses from a pool of likely options. This makes AI-generated search results inherently unpredictable and variable over time.
A prompt does not issue a command to an LLM like code to a computer. Instead, it activates relevant patterns within the model's weight space, guiding it to generate a completion consistent with its training data for having followed such instructions.
Modern chatbot UIs obscure crucial model uncertainty by presenting single, definitive token generations. Bigelow advocates moving away from retro command-line-style chat boxes toward graphical interfaces that reveal token log probabilities, branching alternative completions, and the exact junctures where decision uncertainty collapsed.