A "roofline analysis" reveals that LLM performance is limited by the slower of two factors: the time it takes to fetch model parameters from memory (memory-bound) or the time it takes to perform matrix multiplications (compute-bound). Optimizing performance requires identifying and addressing the correct bottleneck.

While faster model versions like Opus 4.6 Fast offer significant speed improvements, they come at a steep cost—six times the price of the standard model. This creates a new strategic layer for developers, who must now consciously decide which tasks justify the high expense to avoid unexpectedly large bills.

The necessity of batching stems from a fundamental hardware reality: moving data is far more energy-intensive than computing with it. A single parameter's journey from on-chip SRAM to the multiplier can cost 1000x more energy than the multiplication itself. Batching amortizes this high data movement cost over many computations.

APIs charge less for input prompts (prefill) than for generating responses (decode). This is because prefill processes many tokens at once, becoming compute-bound. Decode generates tokens one-by-one, making each step dominated by the high, unamortized cost of memory access. The price difference reflects this efficiency gap.

The Chinchilla scaling law optimizes pre-training compute alone. However, production models must also account for inference costs. By training smaller models on much more data (~100x the Chinchilla optimum), labs create models that are cheaper to run for users, effectively amortizing the higher training cost over the model's lifetime.

For any given hardware, there is a fundamental lower bound on inference latency. This "latency floor" is the time required to load the model's total parameters from memory (e.g., HBM) onto the chip. This process cannot be sped up by reducing batch size or other software tricks.

The ideal batch size that balances memory-bound and compute-bound operations can be calculated by a simple formula. It's roughly 300 (a hardware constant for modern GPUs) multiplied by the model's sparsity (total parameters / active parameters), providing a practical starting point for performance optimization.

For tasks that don't require immediate results, like generating a day's worth of social media content, using batch processing APIs is a powerful cost-saving measure. It allows agents to queue up and execute large jobs at a fraction of the price of real-time generation.

Parser's AI costs are lower than its server costs. They achieve this by intentionally avoiding the most powerful, expensive LLMs which are often slow and rate-limited. Instead, they find a balance, prioritizing speed and cost-effectiveness to process high volumes affordably.