Separating inference into "prefill" (memory-bound) and "decode" (bandwidth-bound) tasks is a game-changer for hardware longevity. It allows older GPUs to be used for prefill tasks indefinitely, extending their useful economic life from 3-4 years to 10-15 years, a boon for data centers and their financiers.

A core challenge in physical AI is the tension between large, powerful models (offboard, in a data center) and the need for low-latency models (onboard, on the machine). The key is using techniques like distillation to create smaller derivatives that run in milliseconds for safety-critical decisions.

The most in-demand skill at labs like Google DeepMind is low-level engineering for accelerating LLM runtime. This involves creating efficient, custom software artifacts (kernels) for new neural net architectures and serving techniques at scale.

Spreading a model's layers across multiple GPU racks (pipeline parallelism) is a strategy to overcome memory capacity limits on a single rack. However, for inference, it offers no latency improvement; the total time remains the same. Its sole benefit is in memory capacity management for enormous models.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

The model uses a Mixture-of-Experts (MoE) architecture with over 200 billion parameters, but only activates a "sparse" 10 billion for any given task. This design provides the knowledge base of a massive model while keeping inference speed and cost comparable to much smaller models.

Google's focus on fast, cost-effective models like Gemini 3.5 Flash is driven by the needs of its massive-scale products (e.g., Search). For billions of users, low latency and cost are more critical than absolute peak performance, as users are often unwilling to wait for a slightly smarter but slower response.

The public-facing models from major labs are likely efficient Mixture-of-Experts (MOE) versions distilled from much larger, private, and computationally expensive dense models. This means the model users interact with is a smaller, optimized copy, not the original frontier model.

Mixture-of-Experts (MoE) models require an "all-to-all" communication pattern. This is efficient within a single GPU rack's high-speed interconnect but becomes a major bottleneck between racks, where communication is ~8x slower. This effectively limits an MoE layer's maximum size to what a single rack can support.