Because music is subjective, AI music models can't be trained on "right" answers like chess or code. Instead of aiming for peak performance in one genre, Suno's team focuses on identifying and improving areas where the model underperforms, or has "anti-spikes."

Descript's AI audio tool worsened after they trained it on extremely bad audio (e.g., vacuum cleaners). They learned the model that best fixes terrible audio is different from the one that best improves merely "okay" audio—the more common user scenario. You must train for your primary user's reality, not the worst possible edge case.

While text generation has largely converged on the Transformer architecture, the audio AI domain has no single winning recipe. This lack of a settled standard makes the field highly experimental and exciting for researchers exploring novel approaches like diffusion and flow matching.

Unlike LLMs, where performance often scales with size, specific voice AI applications appear to have an optimal parameter count. For tasks like audiobook narration, ElevenLabs believes it has found the size sweet spot, where making models larger yields diminishing returns on quality, suggesting different scaling laws for specialized AI.

While text-based AI models struggle with non-English languages, the problem is exponentially worse for audio models. The lack of diverse, high-quality audio training data (across ages, genders, topics) in various languages is a critical bottleneck for companies aiming for global adoption of audio-first AI.

AI performance engineer Chris Fregley warns that developing on local machines or even consumer-grade GPUs is a waste of time. Critical differences in hardware, memory bandwidth, and drivers mean that accurate profiling and optimization can only be done on the exact production systems, like NVIDIA's Blackwell or Hopper GPUs.

The popular PyTorch Profiler only shows the 'tip of the iceberg.' To achieve meaningful performance gains, engineers must move beyond it and analyze 50-60 low-level GPU metrics related to streaming multiprocessors, instruction pipelines, and specialized function units. Most of the PyTorch community stops too early.

The system offers two tokenizer options: 25 Hz for high-detail audio and 12 Hz for faster generation. This practical approach acknowledges that different applications have different needs, prioritizing either computational efficiency or acoustic fidelity rather than forcing a one-size-fits-all solution.

The optimization layer in DSPy acts like a compiler. Its primary role is to bridge the gap between a developer's high-level, model-agnostic intent and the specific incantations a model needs to perform well. This allows the core program logic to remain clean and portable.