Significant opportunity exists in re-architecting how AI models work. Instead of building ever-larger single models, the focus is shifting to creating networks of smaller, specialized models that collaborate, which can drastically reduce the cost per token produced.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

Liquid AI uses an automated system to discover neural architectures, avoiding human bias. Crucially, it bypasses misleading proxy metrics like perplexity by putting the target hardware in the loop and evaluating models directly on the customer's downstream tasks, optimizing for latency, memory, and quality.

The distinction between a model's architecture and its optimizer is an illusion. Both are learning processes compressing a flow of context—the architecture compresses tokens, while the optimizer compresses gradients. This unified view allows for designing them as one interconnected system.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

AI development is inefficiently split into pre-training (optimizing for compression) and RL (optimizing for tasks), where RL often invalidates pre-training metrics. Combining these into a unified, end-to-end learning algorithm focused on final outcomes could yield an order-of-magnitude improvement in training efficiency.

Instead of brute-force training, Roboflow uses Neural Architecture Search (NAS) with weight-sharing. This technique trains thousands of model configurations in a single run, creating a Pareto frontier of options. When run on a custom dataset, it produces a unique "one-of-one" model architecture optimized for that specific problem.

Despite its age, the Transformer architecture is likely here to stay on the path to AGI. A massive ecosystem of optimizers, hardware, and techniques has been built around it, creating a powerful "local minimum" that makes it more practical to iterate on Transformers than to replace them entirely.

A fundamental constraint today is that the model architecture used for training must be the same as the one used for inference. Future breakthroughs could come from lifting this constraint. This would allow for specialized models: one optimized for compute-intensive training and another for memory-intensive serving.