Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Bigelow argues in-context learning encompasses everything a model does to update its representations dynamically outside of weight adjustments. As an LLM parses text, sentences, or its own generated reasoning tokens, its internal latent beliefs undergo non-linear shifts analogous to grokking phase transitions.

Related Insights

The entire deep learning paradigm, including backpropagation, can be viewed as a form of in-context learning. This reframes the pre-training phase not as a separate process, but as the model forming a long-term associative memory, unifying it with inference-time adaptation.

The bottleneck for AI is not raw intelligence but understanding new context. This requires models that continuously learn from new data and interactions, moving beyond the static pre-train/fine-tune paradigm and deeply baking new information into the model weights.

When an LLM is shown few-shot examples of a new task, it is performing Bayesian updating. With each example provided in the prompt, its belief (posterior probability) about the correct next token shifts, allowing it to "learn" a new pattern on the fly without changing its weights.

Contrary to the view that in-context learning is a distinct process from training, Karpathy speculates it might be an emergent form of gradient descent happening within the model's layers. He cites papers showing that transformers can learn to perform linear regression in-context, with internal mechanics that mimic an optimization loop.

Rich Sutton argues that LLM-based assistants are not "experiential learners." Despite in-context learning, their fundamental weights never change after deployment. This prevents them from truly adapting, forming new concepts, or updating their core understanding based on user interactions.

Bigelow predicts that personalized continual learning through weight updates will be bypassed in favor of context engineering and external memory harnesses. Shoving user history and dynamic data directly into runtime context delivers massive customization without risking catastrophic forgetting or representational collapse.

Research suggests a formal equivalence between modifying a model's internal activations (steering) and providing prompt examples (in-context learning). This framework could potentially create a formula to convert between the two techniques, even for complex behaviors like jailbreaks.

While prompt engineering is the interface, context engineering is the "magic" for production systems. It involves strategically managing what information (session history, knowledge base) fits into the model's limited context window. This art directly impacts both cost and performance.

Instead of just expanding context windows, the next architectural shift is toward models that learn to manage their own context. Inspired by Recursive Language Models (RLMs), these agents will actively retrieve, transform, and store information in a persistent state, enabling more effective long-horizon reasoning.

During use (inference), an LLM's weights are frozen. Prompts and context can steer the model's predictions for a single request, but they do not permanently 'teach' it or alter its underlying parameters. This fundamental concept explains why context must be provided repeatedly and clarifies that hallucinations are plausible outputs based on training patterns, not new knowledge.