Liquid AI's origins lie in MIT research modeling the nervous system of the C. elegans worm. This led to differential equation-based networks where a small number of complex "liquid neurons" could perform complex robotics tasks like autonomous driving, showcasing extreme efficiency.
Liquid AI's early, highly effective non-linear models faced a major scaling bottleneck. Non-linear relationships are difficult to "tensorize"—convert from sequential to parallel computations—which is essential for GPU efficiency. This is why linear systems like state-space models scale more easily.
A major breakthrough for Liquid AI was finding a closed-form solution for the differential equations governing their neural networks, a problem unsolved since 1907. This eliminated the need for slow, step-by-step numerical solvers, enabling a massive leap in scalability from hundreds to potentially billions of neurons.
As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.
Liquid AI uses an automated system to discover neural architectures, avoiding human bias. Crucially, it bypasses misleading proxy metrics like perplexity by putting the target hardware in the loop and evaluating models directly on the customer's downstream tasks, optimizing for latency, memory, and quality.
Input-dependent gating, where a model's transformation is modified based on the current input, is a powerful theme in architectures like Mamba and Liquid AI's LFM. This allows the model to learn adaptable *dynamics* during backpropagation, not just static parameters, which is key for efficiency and generalization.
According to Liquid AI's CEO, the primary application of architectural research has become enabling efficiency—reducing cost, latency, and memory without sacrificing quality. The next major breakthroughs in AI *capability* are more likely to stem from new learning algorithms and data paradigms rather than architecture alone.
Liquid AI's automated architecture search found that the winning formula for efficient models on CPUs is a hybrid approach. It combines a few attention layers for generalization with a majority of layers made from a simple, double-gated 1D convolution. This balances power and efficiency for on-device AI.
To remain competitive, chip makers like AMD and Qualcomm must evolve beyond optimizing low-level kernels. The new battleground is a vertically integrated "intelligence layer"—offering their own highly-optimized foundation models tailored to their hardware. This strategy, pioneered by Nvidia with its NeMo framework, simplifies enterprise adoption.
The quadratic scaling of attention is a bottleneck for extremely long sequences. For specific domains like genomics, which involve massive sequences but a very small vocabulary (e.g., A, C, G, T), attention is overkill. More efficient architectures like pure convolutions or state-space models are better suited.
Powerful on-device AI won't be a single large model. The effective paradigm is a smaller "orchestrator" model that acts as a router. It handles simple tasks, calls specialized local models (e.g., for PII filtering), and intelligently decides when to escalate complex queries to more powerful cloud-based models.
Current AI's in-context learning is an emergent, but limited, form of gradient descent. Ramin Hasani argues that human intelligence is far more sophisticated, emerging from a diverse toolkit of learning algorithms like reinforcement learning and Bayesian reasoning running "in-context." Achieving human-level intelligence requires discovering how to elicit these other algorithms.