The success of neural networks on problems like Go and protein folding, long considered intractable NP-hard problems, is profound. It suggests our formal understanding of computational hardness, which focuses on worst-case scenarios, may be an incomplete model for how to find useful, approximate solutions in practice.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

A quaternion network showed superior extrapolation, suggesting unique representational power. However, analysis revealed the equivalent real-valued network could find the solution; the quaternion parameterization just created an "optimization basin" that SGD found more reliably. The advantage was in optimization ease, not fundamental capability.

The distinction between a model's architecture and its optimizer is an illusion. Both are learning processes compressing a flow of context—the architecture compresses tokens, while the optimizer compresses gradients. This unified view allows for designing them as one interconnected system.

An internal, general-purpose OpenAI model solved a famous combinatorial geometry problem without specialized training or scaffolding. Unlike task-specific AIs, this achievement demonstrates a significant advance in abstract reasoning, suggesting models are progressing towards more general intelligence faster than anticipated.

The fundamental primitive for AI chips isn't arbitrary; it's the multiply-accumulate (MAC) operation. This is because it directly maps to the innermost computational loop of matrix multiplication (output += input1 * input2), which is the foundational computation for most neural networks.

Biohub's goal was to create a general world model that "understands proteins." An emergent property of this generalist model was state-of-the-art performance in the highly specialized task of designing single-chain antibodies, a critical function for therapeutics. This demonstrates the power of general models to solve niche problems without explicit training.

Many papers claim exotic algebras (Quaternion, etc.) are parameter-efficient. However, they often fail to compare against real-valued networks with equivalent structural constraints. When properly matched controls are introduced, the supposed advantage from the exotic algebra evaporates, revealing the benefit was simply from the imposed structure itself, not the algebra.

Contrary to trends in other AI fields, structural biology problems are not yet dominated by simple, scaled-up transformers. Specialized architectures that bake in physical priors, like equivariance, still yield vastly superior performance, as the domain's complexity requires strong inductive biases.