Powerful on-device AI won't be a single large model. The effective paradigm is a smaller "orchestrator" model that acts as a router. It handles simple tasks, calls specialized local models (e.g., for PII filtering), and intelligently decides when to escalate complex queries to more powerful cloud-based models.

While often discussed for privacy, running models on-device eliminates API latency and costs. This allows for near-instant, high-volume processing for free, a key advantage over cloud-based AI services.

A core challenge in physical AI is the tension between large, powerful models (offboard, in a data center) and the need for low-latency models (onboard, on the machine). The key is using techniques like distillation to create smaller derivatives that run in milliseconds for safety-critical decisions.

A significant number of popular articles focus on deploying models using TensorFlow Lite for mobile and other frameworks for web browsers. This signals a major trend towards running AI on user devices, reducing latency and reliance on cloud infrastructure for real-time applications.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

While speed benchmarks are flashy, a model's memory usage is the true determinant of its viability. In real-world applications, AI models must share limited resources with other processes, making a low memory footprint more critical than a marginal speed advantage for successful deployment.

Companies like OpenAI and Anthropic are intentionally shrinking their flagship models (e.g., GPT-4.0 is smaller than GPT-4). The biggest constraint isn't creating more powerful models, but serving them at a speed users will tolerate. Slow models kill adoption, regardless of their intelligence.

Contrary to the idea that infrastructure problems get commoditized, AI inference is growing more complex. This is driven by three factors: (1) increasing model scale (multi-trillion parameters), (2) greater diversity in model architectures and hardware, and (3) the shift to agentic systems that require managing long-lived, unpredictable state.

Inference engineering is not a monolith. Data center teams focus on making models "less slow" for massive throughput. Local AI teams focus on making models "less dumb" on constrained hardware, using methods like advanced quantization to fit models in memory.