The entire deep learning paradigm, including backpropagation, can be viewed as a form of in-context learning. This reframes the pre-training phase not as a separate process, but as the model forming a long-term associative memory, unifying it with inference-time adaptation.

A self-referential or self-modifying model, which generates its own update values based on its current state and inputs, is more powerful than a static one. This process is akin to 'learning how to learn,' allowing for greater adaptability and performance on sequential reasoning tasks.

Contrary to the view that in-context learning is a distinct process from training, Karpathy speculates it might be an emergent form of gradient descent happening within the model's layers. He cites papers showing that transformers can learn to perform linear regression in-context, with internal mechanics that mimic an optimization loop.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

A key trend in AI models is "dynamism"—the ability to vary computation and memory usage per token, as seen in Mixture-of-Experts (MoE) architectures. Current hardware, designed before this trend, is inefficient. New chips must be built to accelerate these dynamic computations.

Liquid AI uses an automated system to discover neural architectures, avoiding human bias. Crucially, it bypasses misleading proxy metrics like perplexity by putting the target hardware in the loop and evaluating models directly on the customer's downstream tasks, optimizing for latency, memory, and quality.

A new model architecture allows robots to vary their internal 'thinking' iterations at test time. This lets practitioners trade response speed for decision accuracy on a case-by-case basis, boosting performance on complex tasks without needing to retrain the model.

Liquid AI's automated architecture search found that the winning formula for efficient models on CPUs is a hybrid approach. It combines a few attention layers for generalization with a majority of layers made from a simple, double-gated 1D convolution. This balances power and efficiency for on-device AI.

A major flaw in current AI is that models are frozen after training and don't learn from new interactions. "Nested Learning," a new technique from Google, offers a path for models to continually update, mimicking a key aspect of human intelligence and overcoming this static limitation.