Contrary to the idea that new technologies make old ones obsolete, AI's evolution is a cumulative stack. Each new layer, like deep learning or generative AI, is built upon and extends the capabilities of the one beneath it, all the way down to the principles of classical AI.

Demis Hassabis argues against an LLM-only path to AGI, citing DeepMind's successes like AlphaGo and AlphaFold as evidence. He advocates for "hybrid systems" (or neurosymbolics) that combine neural networks with other techniques like search or evolutionary methods to discover truly new knowledge, not just remix existing data.

AI and formal methods have been separate fields with opposing traits: AI is flexible but untrustworthy, while formal methods offer guarantees but are rigid. The next frontier is combining them into neurosymbolic systems, creating a "peanut butter and chocolate" moment that captures the best of both worlds.

Instead of relying solely on 'black box' LLMs, a more robust approach is neurosymbolic computation. This method combines three estimators: a traditional symbolic/rule-based model (e.g., a medical checklist), a neural network prediction, and an LLM's assessment. By comparing these diverse outputs, experts can make more informed and reliable judgments.

Purely sequence-based prediction models, while powerful, have fundamental limitations in understanding causality. Achieving robust, trustworthy AI will likely require a hybrid approach that integrates current transformer architectures with symbolic systems, world models, and dedicated causal reasoning components.

While LLMs regress to the mean, neurosymbolic models are 1/1000th the training cost, update in real-time without retraining, and offer the explainability required for high-trust applications like e-commerce search where subjective "taste" matters.

Breakthroughs will emerge from 'systems' of AI—chaining together multiple specialized models to perform complex tasks. GPT-4 is rumored to be a 'mixture of experts,' and companies like Wonder Dynamics combine different models for tasks like character rigging and lighting to achieve superior results.

Modern AI agents, which wrap a large language model in a broader cognitive architecture for decision-making, are not a new concept. They mirror the structure of "expert systems" from the 1980s, which built similar architectures around a core of human-programmed if-then rules instead of a neural network.

The computer industry originally chose a "hyper-literal mathematical machine" path over a "human brain model" based on neural networks, a theory that existed since the 1940s. The current AI wave represents the long-delayed success of that alternate, abandoned path.