"Supporting" a new model requires extensive engineering: re-doing quantization, training new speculative decoders, and adapting to novel architectures. This often kicks off a public race among providers to achieve the highest tokens-per-second.

For specialized, high-stakes tasks like real-time AI policy enforcement, a custom-trained Small Language Model (SLM) can be superior to a general frontier model. Rubrik's SAGE SLM achieved higher accuracy and 5x faster processing by optimizing for performance, cost, and low latency.

For long queries, Baseten first checks for cached inputs. It then uses disaggregated GPUs—one set for pre-fill (processing input) and another for decode (generating tokens), often with a speculative decoder to accelerate output.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

For most enterprise tasks, massive frontier models are overkill—a "bazooka to kill a fly." Smaller, domain-specific models are often more accurate for targeted use cases, significantly cheaper to run, and more secure. They focus on being the "best-in-class employee" for a specific task, not a generalist.

The gap between a basic and a highly optimized inference setup is massive. Stacking techniques like quantization, custom speculative decoders, and KV-aware routing can yield performance improvements of 4x to 10x over a standard baseline for the same model and hardware.

An emerging rule from enterprise deployments is to use small, fine-tuned models for well-defined, domain-specific tasks where they excel. Large models should be reserved for generic, open-ended applications with unknown query types where their broad knowledge base is necessary. This hybrid approach optimizes performance and cost.

While frontier models like Claude excel at analyzing a few complex documents, they are impractical for processing millions. Smaller, specialized, fine-tuned models offer orders of magnitude better cost and throughput, making them the superior choice for large-scale, repetitive extraction tasks.

The biggest performance gains in LLM inference come from speculative decoding, which uses a smaller model to predict tokens in batches. This provides a multiplicative speedup, while optimizing low-level kernels only yields marginal, percentage-point improvements.