Unlike LLMs, parameter count is a misleading metric for AI models in structural biology. These models have fewer than a billion parameters but are more computationally expensive to run due to cubic operations that model pairwise interactions, making inference cost the key bottleneck.

Attention can be understood as an update module with an infinite frequency. It acts as a perfect cache, accessing the entire context at once. However, this is also its weakness: it lacks an inherent understanding of temporal dependency and sequential reasoning, requiring positional encodings as a crutch.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

Current AI models become exponentially more expensive as input size grows (quadratic scaling). New "subquadratic" architectures, however, scale linearly by pre-selecting relevant data. This change could slash compute costs by orders of magnitude, making massive context windows economically viable.

Contrary to trends in other AI fields, structural biology problems are not yet dominated by simple, scaled-up transformers. Specialized architectures that bake in physical priors, like equivariance, still yield vastly superior performance, as the domain's complexity requires strong inductive biases.

Autoencoding models (e.g., BERT) are "readers" that fill in blanks, while autoregressive models (e.g., GPT) are "writers." For non-generative tasks like classification, a tiny autoencoding model can match the performance of a massive autoregressive one, offering huge efficiency gains.

For board games like Go, ResNet architectures can outperform Transformers in lower-data regimes. ResNets have a built-in inductive bias for local spatial patterns via convolutions, which is highly relevant for Go. Transformers must learn these patterns from scratch, requiring more data to achieve similar performance.

A major misconception is that general-purpose Large Language Models (LLMs) can be readily applied to complex biological problems. Biological data, like RNA sequencing, constitutes a unique language that requires custom-built foundation models, not simply fine-tuning of existing LLMs.

Liquid AI's automated architecture search found that the winning formula for efficient models on CPUs is a hybrid approach. It combines a few attention layers for generalization with a majority of layers made from a simple, double-gated 1D convolution. This balances power and efficiency for on-device AI.