The quality and vision of an AI-generated video are determined more by the source reference images and videos than by the text prompt itself. Providing a strong visual reference gives the model a clear understanding of taste, style, and desired outcome, acting as a more powerful input than descriptive text alone.

The perceived intelligence of video generation models is often an illusion. The heavy lifting is done by a large language model that rewrites simple user prompts into highly detailed scenes. The video diffusion model itself is less intelligent, simply executing these detailed instructions literally.

The computational requirements for generative media scale dramatically across modalities. If a 200-token LLM prompt costs 1 unit of compute, a single image costs 100x that, and a 5-second video costs another 100x on top of that—a 10,000x total increase. 4K video adds another 10x multiplier.

The Sora team views video as having lower "intelligence per bit" compared to text. However, the total volume of available video data is vastly larger and less tapped. This suggests that, unlike LLMs facing a data crunch, video models can scale with more data for a very long time.

Advanced model training is not just about scraping the web. It's a multi-stage process that starts with massive web data, is refined by human-created examples and ratings (SFT), and is then scaled using reinforcement learning on data generated by the model itself. This synthetic data loop is now a critical component.

Traditional video models process an entire clip at once, causing delays. Descartes' Mirage model is autoregressive, predicting only the next frame based on the input stream and previously generated frames. This LLM-like approach is what enables its real-time, low-latency performance.

Video models are bootstrapped from image models because the denser, cheaper language-to-image data provides a stronger foundation for understanding human intent, a prerequisite for complex video generation.

To analyze video cost-effectively, Tim McLear uses a cheap, fast model to generate captions for individual frames sampled every five seconds. He then packages all these low-level descriptions and the audio transcript and sends them to a powerful reasoning model. This model's job is to synthesize all the data into a high-level summary of the video.

The primary challenge in creating stable, real-time autoregressive video is error accumulation. Like early LLMs getting stuck in loops, video models degrade frame-by-frame until the output is useless. Overcoming this compounding error, not just processing speed, is the core research breakthrough required for long-form generation.