The AI industry is hitting data limits for training massive, general-purpose models. The next wave of progress will likely come from creating highly specialized models for specific domains, similar to DeepMind's AlphaFold, which can achieve superhuman performance on narrow tasks.

The enormous compute budget for the original AlphaGo was not about finding the most efficient training method, but about proving a method could work at all. Once a breakthrough is made and the path is clear, subsequent efforts can focus on optimization and achieve similar results with far less compute.

Go's search space is larger than the number of atoms in the universe, making exhaustive search impossible. AlphaGo's core breakthrough was using neural networks to intelligently guide its search, evaluating only the most promising moves and making an intractable problem solvable.

AlphaGo's architecture mimicked human cognition by pairing a 'fast thinking' neural network for intuition with a 'slow thinking' search algorithm for explicit planning. This hybrid model, combining pattern recognition with calculation, proved more powerful for tackling complex problems than either approach alone.

While the theories behind neural networks existed for decades, their practical application was infeasible. The true catalyst wasn't a new algorithm, but the parallel processing power of GPUs and the availability of massive datasets, which finally made training complex models a reality.

The "bitter lesson" in AI research posits that methods leveraging massive computation scale better and ultimately win out over approaches that rely on human-designed domain knowledge or clever shortcuts, favoring scale over ingenuity.

A core legacy of AlphaGo is turning complex search problems into 'games' for AI agents. AlphaTensor reframed the challenge of finding the fastest matrix multiplication algorithm as a game, allowing it to discover a more efficient method than any human had found in over 50 years, proving the approach's power for scientific discovery.

Humans stop analyzing a game when they intuit a winning or losing position. AlphaGo’s value function mimics this by predicting the eventual outcome from any board state. This allows the search to be drastically shortened, as it doesn't need to play out every possibility to the very end.

A key insight from AlphaGo is that a relatively shallow neural network can approximate the result of an incredibly deep and complex search tree. This suggests neural nets can learn to compress sequential, recursive computation into a single, efficient forward pass.