Voice-to-text services often fail at transcribing voicemails not because of compute limitations, but because they don't use context. They process audio in a vacuum, failing to recognize the recipient's name or other contextual clues that a human—or a smarter AI—would use for accurate interpretation.

While direct speech-to-speech models are faster (lower latency), they are less reliable and "dumber." ElevenLabs bets on a "cascaded" approach that uses text as an intermediate layer, providing greater accuracy, visibility, and control—features that are critical for most enterprise applications.

Unlike LLMs, where performance often scales with size, specific voice AI applications appear to have an optimal parameter count. For tasks like audiobook narration, ElevenLabs believes it has found the size sweet spot, where making models larger yields diminishing returns on quality, suggesting different scaling laws for specialized AI.

A non-obvious failure mode for voice AI is misinterpreting accented English. A user speaking English with a strong Russian accent might find their speech transcribed directly into Russian Cyrillic. This highlights a complex, and frustrating, challenge in building robust and inclusive voice models for a global user base.

The primary driver for fine-tuning isn't cost but necessity. When applications like real-time voice demand low latency, developers are forced to use smaller models. These models often lack quality for specific tasks, making fine-tuning a necessary step to achieve production-level performance.

Early voice models required hardcoding parameters like accent or emotion. Modern models, like those from ElevenLabs, learn these nuances contextually from data, allowing complex traits like a specific accent to emerge naturally without being explicitly programmed.

To solve the problem that enterprise customers don't know how to choose a "good" voice, ElevenLabs created the role of a "voice sommelier." This expert voice coach works with clients to find the right voice for their brand and use case, effectively productizing the subjective process of voice selection and turning it into a sales asset.

Despite base models improving, they only achieve ~90% accuracy for specific subjects. Enterprises require the 99% pixel-perfect accuracy that LoRAs provide for brand and character consistency, making it an essential, long-term feature, not a stopgap solution.

The company needed a high-quality speech-to-text model to annotate its own training data because existing market solutions were inadequate. This internal necessity evolved into a successful, customer-facing product, demonstrating the value of building tools to solve your own critical problems.