Most teams focus on LLM and TTS speed to reduce voice agent latency. However, the biggest improvements come from optimizing end-of-turn detection, moving from fixed silence timeouts to intelligent models that analyze intonation and cadence, often saving 300-500ms per turn.

While Genspark's calling agent can successfully complete a task and provide a transcript, its noticeable audio delays and awkward handling of interruptions highlight a key weakness. Current voice AI struggles with the subtle, real-time cadence of human conversation, which remains a barrier to broader adoption.

Integrating generative AI into Alexa was complex due to its massive scale: hundreds of millions of users, diverse devices, and millions of existing functions. The challenge was weaving the new tech into this landscape without disrupting the user experience, not just adding an LLM.

While direct speech-to-speech models are faster (lower latency), they are less reliable and "dumber." ElevenLabs bets on a "cascaded" approach that uses text as an intermediate layer, providing greater accuracy, visibility, and control—features that are critical for most enterprise applications.

Prioritizing speech-to-text speed over accuracy is a false economy. An inaccurate transcript forces the user into a correction loop, adding round trips of the entire pipeline. This additional latency from user corrections obliterates any milliseconds saved by a faster, less accurate model.

While total generation time might be similar to API calls, local models offer a superior user experience by starting responses almost immediately. This eliminates the unpredictable network latency and random slowdowns common with APIs, making the interaction feel smoother and more reliable.

Despite incredible advances, everyday voice experiences (like on phones or in cars) feel dated. The lag isn't due to technology but a "deployment gap," where large companies are slow to integrate the latest models into consumer hardware and software, creating a disconnect between what's possible and what's available.

A 2-second delay is acceptable for a single user prompt. However, in an agentic system where 20 agents communicate sequentially, that delay compounds to 40 seconds, rendering the application unusable. This shift necessitates infrastructure with sub-second response times, driving hardware deployment to urban centers.

Focusing on average (P50) latency is misleading because users' perception is shaped by the worst interactions, not the typical ones. A single long delay can ruin the experience. Instrumenting and optimizing for tail latency (P90, P95) at each pipeline stage is critical for creating a consistently responsive agent.