We scan new podcasts and send you the top 5 insights daily.
Trey Courtney says Mood began using ElevenLabs three or four years ago when voice quality was only okay, because custom store messages took three or four days with human script approval and voice talent. Early adoption let the company learn the workflow before quality matured.
A one-size-fits-all AI voice fails. For a Japanese healthcare client, ElevenLabs' agent used quick, short responses for younger callers but a calmer, slower style for older callers. This personalization of delivery, not just content, based on demographic context was critical for success.
While direct speech-to-speech models are faster (lower latency), they are less reliable and "dumber." ElevenLabs bets on a "cascaded" approach that uses text as an intermediate layer, providing greater accuracy, visibility, and control—features that are critical for most enterprise applications.
To solve for emotional intelligence in voice AI, ElevenLabs invests in long-term data annotation. They employ over 1,000 former voice coaches and musicians to label qualitative aspects of audio—the 'how' (emotion, style), not just the 'what' (words). This creates a proprietary dataset that is a significant long-term competitive advantage.
ElevenLabs' defense against giants isn't just a better text-to-speech model. Their strategy focuses on building deep, workflow-specific platforms for agents and creatives. This includes features like CRM integrations and collaboration tools, creating a sticky application layer that a foundational model alone cannot replicate.
While the market focused on crypto and metaverse, ElevenLabs targeted audio. They saw it as an overlooked domain with fewer researchers and smaller model sizes, allowing them to build a frontier model without needing billions in initial capital. This strategic niche selection was key to their early success.
The first AI voice models used by DittoDub were highly unstable, often producing gibberish after 15 seconds. To ensure quality, the team had to build secondary 'demon catcher' AIs whose sole job was to monitor the output for errors and alert the founders, even at 3 AM.
The team's breakthrough moment wasn't perfect voice replication, but when their AI model first laughed. They realized that human-like imperfections—laughter, pauses, "ums"—were the critical elements that made the user experience feel genuinely human and believable, leading to their first viral moment on Hacker News.
AI voice isn't just about cost savings. The technology has improved so much that it often provides a better customer experience (NPS) than human agents. This dual benefit of high ROI and improved experience means customers are eagerly adopting these solutions, creating a powerful market pull for founders.
The company needed a high-quality speech-to-text model to annotate its own training data because existing market solutions were inadequate. This internal necessity evolved into a successful, customer-facing product, demonstrating the value of building tools to solve your own critical problems.
Don't wait for perfect infrastructure like APIs or Model Context Protocol (MCP). Winning AI companies, particularly in voice, are building "interim" solutions that work today to solve a deeply broken user experience. The strategic challenge is then navigating from this interim approach to a more durable, long-term model.