AI models improve not just by getting bigger ("scaling laws"), but by adding distinct new capabilities. Recent breakthroughs include the ability to reason through problems (showing their work), use tools like the internet, and process multiple media types like text, images, and audio simultaneously.

AlphaGo's architecture mimicked human cognition by pairing a 'fast thinking' neural network for intuition with a 'slow thinking' search algorithm for explicit planning. This hybrid model, combining pattern recognition with calculation, proved more powerful for tackling complex problems than either approach alone.

An LLM shouldn't do math internally any more than a human would. The most intelligent AI systems will be those that know when to call specialized, reliable tools—like a Python interpreter or a search API—instead of attempting to internalize every capability from first principles.

AI's strength lies in solving "differentiable" problems where being "close enough" is acceptable, like generating an image. Classical code is better for non-differentiable tasks requiring exact precision, like arithmetic or hashing. This framework helps architects decide where to deploy AI versus traditional algorithms.

The path to robust AI applications isn't a single, all-powerful model. It's a system of specialized "sub-agents," each handling a narrow task like context retrieval or debugging. This architecture allows for using smaller, faster, fine-tuned models for each task, improving overall system performance and efficiency.

Don't default to the most powerful AI. A better architectural principle is to give each task the minimum model capability required to solve it reliably. This means knowing when a simple program is better than an SLM, or an SLM is better than a large model.

AI expert Gary Marcus reveals that advanced AI models are not pure neural networks. They incorporate classical, rules-based symbolic AI to add guardrails, check for errors, and improve reliability—a hybrid model he advocated for over two decades ago. The industry is reluctant to acknowledge this shift.

AI and formal methods have been separate fields with opposing traits: AI is flexible but untrustworthy, while formal methods offer guarantees but are rigid. The next frontier is combining them into neurosymbolic systems, creating a "peanut butter and chocolate" moment that captures the best of both worlds.

Breakthroughs will emerge from 'systems' of AI—chaining together multiple specialized models to perform complex tasks. GPT-4 is rumored to be a 'mixture of experts,' and companies like Wonder Dynamics combine different models for tasks like character rigging and lighting to achieve superior results.