Unlike simple classification (one pass), generative AI performs recursive inference. Each new token (word, pixel) requires a full pass through the model, turning a single prompt into a series of demanding computations. This makes inference a major, ongoing driver of GPU demand, rivaling training.

Diffusion models work on a continuous medium like an image by adding noise until it's unrecognizable, then training a model to reverse the process. This holistic, denoising method is fundamentally different from autoregressive models like large language models, which predict data one token at a time.

Spreading a model's layers across multiple GPU racks (pipeline parallelism) is a strategy to overcome memory capacity limits on a single rack. However, for inference, it offers no latency improvement; the total time remains the same. Its sole benefit is in memory capacity management for enormous models.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

Autoregressive models must complete an output before it can be evaluated against constraints. Diffusion's iterative, coarse-to-fine process allows for applying reward functions or constraints *during* generation, enabling more precise control and alignment.

The primary performance bottleneck for LLMs is memory bandwidth (moving large weights), making them memory-bound. In contrast, diffusion-based video models are compute-bound, as they saturate the GPU's processing power by simultaneously denoising tens of thousands of tokens. This represents a fundamental difference in optimization strategy.

The biggest performance gains in LLM inference come from speculative decoding, which uses a smaller model to predict tokens in batches. This provides a multiplicative speedup, while optimizing low-level kernels only yields marginal, percentage-point improvements.

Unlike text, gene expression levels lack inherent order. Autoregressive models (like GPT) force an artificial sequence, limiting performance. Diffusion models, which operate on sets and iteratively refine predictions, are a more natural and effective architecture for modeling cellular responses to perturbations.

GPUs are ill-suited for generating output tokens because they process models in discrete chunks called "kernels." This method requires constant data transfer to and from external memory, creating a significant bottleneck limited by memory bandwidth, which ultimately slows down real-time inference.