"Supporting" a new model requires extensive engineering: re-doing quantization, training new speculative decoders, and adapting to novel architectures. This often kicks off a public race among providers to achieve the highest tokens-per-second.

For long queries, Baseten first checks for cached inputs. It then uses disaggregated GPUs—one set for pre-fill (processing input) and another for decode (generating tokens), often with a speculative decoder to accelerate output.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

OpenAI achieved a major reduction in the cost of running its models through purely software and algorithmic improvements, such as quantization and smarter caching. This demonstrates that efficiency innovation can be as impactful as acquiring more hardware, suggesting a path to overcoming compute bottlenecks without relying solely on expensive chips.

Quantization is a compression technique that shrinks AI models to run on weaker hardware with minimal quality loss. Understanding this concept is key, as it effectively allows you to run models that would otherwise require server-grade equipment on a standard laptop, essentially doubling your hardware's capability.

SambaNova's architecture is optimized for inference by treating it as a data movement challenge rather than a raw compute problem. By designing for efficient data flow and communication between memory and compute units, they achieve 5-10x performance improvements over traditional GPUs.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

Inference engineering is not a monolith. Data center teams focus on making models "less slow" for massive throughput. Local AI teams focus on making models "less dumb" on constrained hardware, using methods like advanced quantization to fit models in memory.

The biggest performance gains in LLM inference come from speculative decoding, which uses a smaller model to predict tokens in batches. This provides a multiplicative speedup, while optimizing low-level kernels only yields marginal, percentage-point improvements.