A "roofline analysis" reveals that LLM performance is limited by the slower of two factors: the time it takes to fetch model parameters from memory (memory-bound) or the time it takes to perform matrix multiplications (compute-bound). Optimizing performance requires identifying and addressing the correct bottleneck.

While often discussed for privacy, running models on-device eliminates API latency and costs. This allows for near-instant, high-volume processing for free, a key advantage over cloud-based AI services.

A core challenge in physical AI is the tension between large, powerful models (offboard, in a data center) and the need for low-latency models (onboard, on the machine). The key is using techniques like distillation to create smaller derivatives that run in milliseconds for safety-critical decisions.

Top inference frameworks separate the prefill stage (ingesting the prompt, often compute-bound) from the decode stage (generating tokens, often memory-bound). This disaggregation allows for specialized hardware pools and scheduling for each phase, boosting overall efficiency and throughput.

The critical open-source inference engine VLLM began in 2022, pre-ChatGPT, as a small side project. The goal was simply to optimize a slow demo for Meta's now-obscure OPT model, but the work uncovered deep, unsolved systems problems in autoregressive model inference that took years to tackle.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

Optimizing transformer inference, specifically the separation of pre-fill (KV cache building) and decode (token generation), is becoming a foundational skill. Chris Fregly predicts this complex topic, known as disaggregated pre-fill decode, will be a core component of AI engineering interviews at top labs within two years.

Instead of a single "omni-model," Mistral offers both large, general-purpose models and smaller, highly optimized models for specific tasks like transcription. This allows customers to choose a cost-effective solution for dedicated use cases without paying for unneeded capabilities.

Mistral-Medium-3.5 allows users to adjust its "reasoning effort" per request. This unique feature enables the same model weights to deliver either quick responses for simple queries or perform extended computation for complex agentic tasks, optimizing the trade-off between latency and solution quality.