A useful mental model for an LLM is a giant matrix where each row is a possible prompt and columns represent next-token probabilities. This matrix is impossibly large but also extremely sparse, as most token combinations are gibberish. The LLM's job is to efficiently compress and approximate this matrix.

In the 'word gravity' model, semantically light function words like 'to' can exhibit sharp curvature in the embedding space. Their meaning is highly dependent on context, making them more 'influenceable' and causing their position to shift dramatically from layer to layer compared to more stable words.

Attention can be understood as an update module with an infinite frequency. It acts as a perfect cache, accessing the entire context at once. However, this is also its weakness: it lacks an inherent understanding of temporal dependency and sequential reasoning, requiring positional encodings as a crutch.

A common misconception is that Transformers are sequential models like RNNs. Fundamentally, they are permutation-equivariant and operate on sets of tokens. Sequence information is artificially injected via positional embeddings, making the architecture inherently flexible for non-linear data like 3D scenes or graphs.

Moving beyond the simple Linear Representation Hypothesis, models organize concepts within sparse mixtures of subspaces. The specific geometry of these "manifolds" (e.g., a circle for days of the week) encodes the relationships and valid operations between concepts, like chemistry emerging from the periodic table.

The core transformer architecture is permutation-equivariant and operates on sets of tokens, not ordered sequences. Sequentiality is an add-on via positional embeddings, making transformers naturally suited for non-linear data structures like 3D worlds, a concept many practitioners overlook.

The 'attention' mechanism in AI has roots in 1990s robotics. Dr. Wallace built a robotic eye with high resolution at its center and lower resolution in the periphery. The system detected 'interesting' data (e.g., movement) in the periphery and rapidly shifted its high-resolution gaze—its 'attention'—to that point, a physical analog to how LLMs weigh words.

Contrary to common perception shaped by their use in language, Transformers are not inherently sequential. Their core architecture operates on sets of tokens, with sequence information only injected via positional embeddings. This makes them powerful for non-sequential data like 3D objects or other unordered collections.

Dr. Luis Serrano's research presents a "word gravity" analogy where words in a transformer don't just "pay attention" but physically bend the embedding space, pulling other words along curved paths, much like planets orbiting the sun. This provides a visual, physical intuition for the attention mechanism.