For long queries, Baseten first checks for cached inputs. It then uses disaggregated GPUs—one set for pre-fill (processing input) and another for decode (generating tokens), often with a speculative decoder to accelerate output.
While pay-per-token APIs are great for experimentation, users pushing millions of tokens per hour find it significantly cheaper to rent dedicated hardware ("by the box") and manage saturation themselves. This marks the inflection point for serious production use cases.
For dedicated deployments, a small "draft" model can be trained on traffic-specific data (e.g., Harry Potter books). This allows the draft model to predict subsequent tokens with high accuracy, significantly accelerating the main model's decoding speed for that specific use case.
"Supporting" a new model requires extensive engineering: re-doing quantization, training new speculative decoders, and adapting to novel architectures. This often kicks off a public race among providers to achieve the highest tokens-per-second.
Instead of costly full retraining, engineers are combining parts of different open-source models. For example, Baseten grafted the Kimi vision encoder onto GLM-5.2 by only retraining a small projector layer, adding vision capabilities without degrading original performance.
Bizarre inference bugs, like a model endlessly repeating a token, may not be a model or software issue. They can be elusive race conditions in CUDA kernels, only exposed by specific hardware configurations, such as a slower node-to-node interconnect in one cluster versus another.
Contrary to the belief that more quantization always means more quality loss, it's possible for errors introduced in different quantized layers to cancel each other out. This allows for quantizing a model more aggressively while achieving higher fidelity to the original.
The gap between a basic and a highly optimized inference setup is massive. Stacking techniques like quantization, custom speculative decoders, and KV-aware routing can yield performance improvements of 4x to 10x over a standard baseline for the same model and hardware.
Inference engineering is not a monolith. Data center teams focus on making models "less slow" for massive throughput. Local AI teams focus on making models "less dumb" on constrained hardware, using methods like advanced quantization to fit models in memory.
Modern GPUs like NVIDIA's Rubin are increasingly designed with highly specialized components tailored for transformer architectures. This trend blurs the line between general-purpose GPUs and specialized ASICs, making it harder for standalone AI ASIC companies to compete.
Generating high-quality, long-form video with current diffusion models is computationally unfeasible. The O(n^2) complexity of full attention over tens of thousands of tokens (for just seconds of video) creates a massive bottleneck, forcing a trade-off between quality and length.
The traditional separation is disappearing. Fast inference is critical for modern training (e.g., RL rollouts), while training techniques are now essential for inference optimization (e.g., training speculative decoders). This requires engineers to be proficient in both domains.
In a powerful feedback loop, Baseten used their GLM-5.2 endpoint to analyze its own performance traces, identify bottlenecks in its inference engine, and then write new, more efficient GPU kernels. The model was literally optimizing the stack it was running on.