The SNR-T bias can be fixed efficiently without retraining models. At each denoising step, the image is broken into frequency bands using wavelets. Each band is then given a small correction based on its specific noise mismatch before being recombined. This surgical approach is computationally cheap and universally effective.

Descript's AI audio tool worsened after they trained it on extremely bad audio (e.g., vacuum cleaners). They learned the model that best fixes terrible audio is different from the one that best improves merely "okay" audio—the more common user scenario. You must train for your primary user's reality, not the worst possible edge case.

Unlike simple classification (one pass), generative AI performs recursive inference. Each new token (word, pixel) requires a full pass through the model, turning a single prompt into a series of demanding computations. This makes inference a major, ongoing driver of GPU demand, rivaling training.

Modern protein models use a generative approach (diffusion) instead of regression. Instead of predicting one "correct" structure, they model a distribution of possibilities. This better handles molecular dynamism and avoids averaging between multiple valid states, which is a flaw of regression models.

Diffusion models naturally reconstruct images in layers. In early denoising stages with high noise, they focus on low-frequency information like overall composition and color. As noise decreases in later steps, they add high-frequency details like textures and sharp edges. This hierarchical process is key to understanding their behavior.

A significant hurdle for using large vision models in production is their non-deterministic nature. The same model can produce different results for the same query at different times, making it difficult to build reliable, consistent downstream systems. This unpredictability is a key challenge alongside speed and cost.

Models like Stable Diffusion achieve massive compression ratios (e.g., 50,000-to-1) because they aren't just storing data; they are learning the underlying principles and concepts. The resulting model is a compact 'filter' of intelligence that can generate novel outputs based on these learned principles.

Karpathy warns that training AIs on synthetically generated data is dangerous due to "model collapse." An AI's output, while seemingly reasonable case-by-case, occupies a tiny, low-entropy manifold of the possible solution space. Continual training on this collapsed distribution causes the model to become worse and less diverse over time.

The primary challenge in creating stable, real-time autoregressive video is error accumulation. Like early LLMs getting stuck in loops, video models degrade frame-by-frame until the output is useless. Overcoming this compounding error, not just processing speed, is the core research breakthrough required for long-form generation.