Significant latency in voice agents comes from the handoffs between disparate services (STT, LLM, TTS). Each vendor hop adds network boundaries, round-trip latency, and potential points of failure. Consolidating the pipeline or co-locating services removes these 'seams' and reduces compounded delays.

Most teams focus on LLM and TTS speed to reduce voice agent latency. However, the biggest improvements come from optimizing end-of-turn detection, moving from fixed silence timeouts to intelligent models that analyze intonation and cadence, often saving 300-500ms per turn.

Voice-to-voice AI models promise more natural, low-latency conversations by processing audio directly. However, they are currently impractical for many high-stakes enterprise applications due to a hallucination rate that can be eight times higher than text-based systems.

Success for dictation tools is measured not by raw accuracy, but by the percentage of messages that are perfect and require no manual correction. While incumbents like Apple have a ~10% 'zero edit rate,' Whisperflow's 85% rate is what drives adoption by eliminating the friction of post-dictation fixes.

Voice-to-text services often fail at transcribing voicemails not because of compute limitations, but because they don't use context. They process audio in a vacuum, failing to recognize the recipient's name or other contextual clues that a human—or a smarter AI—would use for accurate interpretation.

While direct speech-to-speech models are faster (lower latency), they are less reliable and "dumber." ElevenLabs bets on a "cascaded" approach that uses text as an intermediate layer, providing greater accuracy, visibility, and control—features that are critical for most enterprise applications.

The primary driver for fine-tuning isn't cost but necessity. When applications like real-time voice demand low latency, developers are forced to use smaller models. These models often lack quality for specific tasks, making fine-tuning a necessary step to achieve production-level performance.

Using speech-to-text to talk to an AI is not just about speed. The 'art of the ramble' allows you to provide messy, uncertain, and richer context that you would filter out when typing. This gives the model access to your unpolished thought process, enabling it to help clarify your thinking and produce better results.

Focusing on average (P50) latency is misleading because users' perception is shaped by the worst interactions, not the typical ones. A single long delay can ruin the experience. Instrumenting and optimizing for tail latency (P90, P95) at each pipeline stage is critical for creating a consistently responsive agent.