An early project used an NVIDIA model trained only on road data. Artists immediately used it to create surreal images like "a million pedestrians," demonstrating that even niche, "boring" models could be repurposed for creative expression. This became a core belief at Runway: give artists tools, and they'll find unexpected uses.
Before releasing its famous generative models, Runway's main product was a tool called Green Screen that automated rotoscoping, an extremely manual post-production task. This practical tool was used in major films like "Everything Everywhere All at Once" and established the company's user base in the creative industry.
Instead of building a direct text-to-video model from scratch, which was difficult, Runway pragmatically combined a new text-to-depth model with their existing Gen 1 (depth-to-video) model. This two-stage pipeline was a clever shortcut to ship a text-to-video product months faster.
The development of camera controls for Runway's Gen 2 model sparked a key realization. Instead of just "creating" a video, users felt like they were "navigating" a 3D world. This subtle shift in user experience was the seed that grew into the company's entire research direction on world models.
Countering Yann LeCun, Runway's cofounder argues that predicting video frames at scale *is* learning world dynamics. As models scale, their ability to simulate physics predictably improves on benchmarks. This aligns with the "bitter lesson" of AI: general methods that leverage computation and data ultimately outperform specialized, human-designed ones.
The shock of Sora's release served as a powerful catalyst for Runway. The external pressure forced the company to rapidly solve internal challenges, scale their model size and compute 10x, and ship a competitive model (Gen3) in a compressed three-month timeframe, a period employees recall as their favorite.
The "Interface World Model" treats software interfaces as real-time video. Instead of coding with HTML/CSS, developers can describe UI behavior in natural language. The model generates the interactive pixels directly, enabling rapid prototyping, exploration, and personalization.
Runway’s robotics thesis is that pre-training on massive, easily available third-person video data (e.g., people performing tasks) is more scalable and effective than relying on expensive, limited teleoperation or first-person data. This general world knowledge can then be fine-tuned for specific robotic tasks.
A world model has succeeded when a user in a VR headset can't distinguish between the headset's real-world "passthrough" camera feed and a fully generated, interactive environment. If you can interact with the world and are unsure if it's real or rendered, the model has passed the test.
The development path for AI models follows a pattern: a new capability (e.g., better prompting, multi-shot editing) is first implemented as a separate "harness" or scaffold around the core model. Over time, this external logic is absorbed directly into the model's architecture and learned end-to-end.