Early AI pioneers modeled neural network algorithms on the hierarchical structure of visual neurons discovered by neuroscientists like Hubel and Wiesel in the 1950s. This direct inspiration from biology was a pivotal starting point for modern AI, bridging neuroscience and computation.

The entire deep learning paradigm, including backpropagation, can be viewed as a form of in-context learning. This reframes the pre-training phase not as a separate process, but as the model forming a long-term associative memory, unifying it with inference-time adaptation.

Today's AI, particularly neural networks, stems from a long tradition in cognitive science where psychologists used mathematical models to understand human thought. Key advances in neural nets were made by researchers trying to replicate how human minds work, not just build intelligent machines.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

A common misconception is that Transformers are sequential models like RNNs. Fundamentally, they are permutation-equivariant and operate on sets of tokens. Sequence information is artificially injected via positional embeddings, making the architecture inherently flexible for non-linear data like 3D scenes or graphs.

The core transformer architecture is permutation-equivariant and operates on sets of tokens, not ordered sequences. Sequentiality is an add-on via positional embeddings, making transformers naturally suited for non-linear data structures like 3D worlds, a concept many practitioners overlook.

The 'attention' mechanism in AI has roots in 1990s robotics. Dr. Wallace built a robotic eye with high resolution at its center and lower resolution in the periphery. The system detected 'interesting' data (e.g., movement) in the periphery and rapidly shifted its high-resolution gaze—its 'attention'—to that point, a physical analog to how LLMs weigh words.

The 2017 introduction of "transformers" revolutionized AI. Instead of being trained on the specific meaning of each word, models began learning the contextual relationships between words. This allowed AI to predict the next word in a sequence without needing a formal dictionary, leading to more generalist capabilities.

The computer industry originally chose a "hyper-literal mathematical machine" path over a "human brain model" based on neural networks, a theory that existed since the 1940s. The current AI wave represents the long-delayed success of that alternate, abandoned path.