A "roofline analysis" reveals that LLM performance is limited by the slower of two factors: the time it takes to fetch model parameters from memory (memory-bound) or the time it takes to perform matrix multiplications (compute-bound). Optimizing performance requires identifying and addressing the correct bottleneck.

Instead of using massive, expensive LLMs for every task, companies can solve the "tokenpocalypse" (runaway token costs) by pairing smaller models with high-quality retrieval systems. This allows cheap models to act like large ones, saving significant costs.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

OpenAI achieved a major reduction in the cost of running its models through purely software and algorithmic improvements, such as quantization and smarter caching. This demonstrates that efficiency innovation can be as impactful as acquiring more hardware, suggesting a path to overcoming compute bottlenecks without relying solely on expensive chips.

The ideal batch size that balances memory-bound and compute-bound operations can be calculated by a simple formula. It's roughly 300 (a hardware constant for modern GPUs) multiplied by the model's sparsity (total parameters / active parameters), providing a practical starting point for performance optimization.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

Optimizing transformer inference, specifically the separation of pre-fill (KV cache building) and decode (token generation), is becoming a foundational skill. Chris Fregly predicts this complex topic, known as disaggregated pre-fill decode, will be a core component of AI engineering interviews at top labs within two years.

The binary distinction between "reasoning" and "non-reasoning" models is becoming obsolete. The more critical metric is now "token efficiency"—a model's ability to use more tokens only when a task's difficulty requires it. This dynamic token usage is a key differentiator for cost and performance.

API providers offer faster inference at a premium by reducing the number of users processed simultaneously (batch size). This lowers latency but makes each token more expensive because the fixed cost of loading model weights is spread across fewer requests, reducing amortization.