The success of neural networks on problems like Go and protein folding, long considered intractable NP-hard problems, is profound. It suggests our formal understanding of computational hardness, which focuses on worst-case scenarios, may be an incomplete model for how to find useful, approximate solutions in practice.

Go's search space is larger than the number of atoms in the universe, making exhaustive search impossible. AlphaGo's core breakthrough was using neural networks to intelligently guide its search, evaluating only the most promising moves and making an intractable problem solvable.

AlphaGo's architecture mimicked human cognition by pairing a 'fast thinking' neural network for intuition with a 'slow thinking' search algorithm for explicit planning. This hybrid model, combining pattern recognition with calculation, proved more powerful for tackling complex problems than either approach alone.

The "Attention is All You Need" paper's key breakthrough was an architecture designed for massive scalability across GPUs. This focus on efficiency, anticipating the industry's shift to larger models, was more crucial to its dominance than the attention mechanism itself.

While acknowledging the power of scale, Moonlake argues that incorporating symbolic structure allows models to learn with orders of magnitude less data. This mirrors human cognition, which uses abstracted semantic descriptions rather than processing every pixel.

Contrary to trends in other AI fields, structural biology problems are not yet dominated by simple, scaled-up transformers. Specialized architectures that bake in physical priors, like equivariance, still yield vastly superior performance, as the domain's complexity requires strong inductive biases.

Humans stop analyzing a game when they intuit a winning or losing position. AlphaGo’s value function mimics this by predicting the eventual outcome from any board state. This allows the search to be drastically shortened, as it doesn't need to play out every possibility to the very end.

To bridge the learning efficiency gap between humans and AI, researchers use meta-learning. This technique learns optimal initial weights for a neural network, giving it a "soft bias" that starts it closer to a good solution. This mimics the inherent inductive biases that allow humans to learn efficiently from limited data.

A key insight from AlphaGo is that a relatively shallow neural network can approximate the result of an incredibly deep and complex search tree. This suggests neural nets can learn to compress sequential, recursive computation into a single, efficient forward pass.