The choice of optimization algorithm dictates which model architectures are viable. While weaker optimizers require simpler models, a more powerful optimizer can successfully train more complex, harder-to-optimize architectures. This shows how optimizer and architecture research are deeply intertwined, unlocking new performance possibilities.

Liquid AI's early, highly effective non-linear models faced a major scaling bottleneck. Non-linear relationships are difficult to "tensorize"—convert from sequential to parallel computations—which is essential for GPU efficiency. This is why linear systems like state-space models scale more easily.

While hard-coding physical symmetries (equivariance) into a model is theoretically efficient, it can fail in practice. Prof. Welling explains that these constraints can complicate the optimization landscape, making it harder to find good minima. Sometimes, abundant data augmentation with a simpler model yields superior results.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

A quaternion network showed superior extrapolation, suggesting unique representational power. However, analysis revealed the equivalent real-valued network could find the solution; the quaternion parameterization just created an "optimization basin" that SGD found more reliably. The advantage was in optimization ease, not fundamental capability.

The structure of neural networks with activation functions like ReLU can be modeled by "threshold circuits" (TC circuits). These circuits use majority gates instead of traditional AND/OR gates, providing a formal framework from complexity theory for analyzing the computational power of neural net architectures.

Contrary to trends in other AI fields, structural biology problems are not yet dominated by simple, scaled-up transformers. Specialized architectures that bake in physical priors, like equivariance, still yield vastly superior performance, as the domain's complexity requires strong inductive biases.

Much RL research from 2015-2022 has not proven useful in practice because academia rewards complex, math-heavy ideas. These provide implicit "knobs" to overfit benchmarks, while ignoring simpler, more generalizable approaches that may lack intellectual novelty.

Today's transformers are optimized for matrix multiplication (MatMul) on GPUs. However, as compute scales to distributed clusters, MatMul may not be the most efficient primitive. Future AI architectures could be drastically different, built on new primitives better suited for large-scale, distributed hardware.