The bottleneck for AI is not raw intelligence but understanding new context. This requires models that continuously learn from new data and interactions, moving beyond the static pre-train/fine-tune paradigm and deeply baking new information into the model weights.

Fine-tuning creates model-specific optimizations that quickly become obsolete. Blitzy favors developing sophisticated, system-level "memory" that captures enterprise-specific context and preferences. This approach is model-agnostic and more durable as base models improve, unlike fine-tuning which requires constant rework.

Venture capitalists are increasingly being pitched by AI startups claiming to have solved "continual learning." However, many of these are simply using clever workarounds, like giving a model a 'scratch pad' to reference new data, rather than building models that can fundamentally learn and update themselves in real-time.

Cohere's Chief AI Officer, Joelle Pineau, finds the concept of continual learning problematic because the research community lacks a universally agreed-upon problem definition, making it difficult to measure progress, unlike more standardized research areas like AI memory.

Instead of relying on opaque model weights, continual learning is more reliably achieved by having AI build explicit, external 'world models' like knowledge graphs. This approach makes the model's understanding inspectable and correctable by humans, enabling more robust causal analysis.

A genuinely continual learner doesn't have separate training and testing phases. Instead, its life is a continuous process divided into two modes: an 'active' phase of interacting with new data and an 'offline' sleep phase for memory consolidation and self-improvement.

The key to a truly intelligent enterprise AI is not a static model, but one that uses reinforcement learning (RL) to continuously update its own weights overnight based on daily interactions, a concept known as 'continuous learning'.

The key to continual learning is not just a longer context window, but a new architecture with a spectrum of memory types. "Nested learning" proposes a model with different layers that update at different frequencies—from transient working memory to persistent core knowledge—mimicking how humans learn without catastrophic forgetting.

A major flaw in current AI is that models are frozen after training and don't learn from new interactions. "Nested Learning," a new technique from Google, offers a path for models to continually update, mimicking a key aspect of human intelligence and overcoming this static limitation.