Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Dmitri Dolgov explains that while open-loop imitation learning suffices for simple or constrained applications, physical systems in uncontrolled environments require a three-part ecosystem: the driver, the simulator, and the critic. Large off-board teacher models distill into an onboard student driver, while realistic simulators provide closed-loop synthetic environments for training and the critic evaluates behavior. In safety-critical contexts, closed-loop training and validation are essential.

Related Insights

The paradigm for training AI has evolved from extracting knowledge from human experts to pass tests. Now, the emphasis is on creating realistic, simulated Reinforcement Learning (RL) environments where AI agents learn by mastering real-world tasks and workflows, much like a pilot in a flight simulator.

In robotics, purely imitating human actions is insufficient. A model trained this way doesn't learn how to recover from inevitable errors. Comma AI solves this by training its models in a simulator where they are forced to learn recovery paths from off-course situations, a critical step for real-world deployment.

Dolgov advises small teams with modest budgets to focus on domains where the environment can be strictly controlled, such as structured warehouses, or to build human-assistive tools. Pursuing fully autonomous systems with superhuman safety demands immense capital to develop world models, realistic simulators, and closed-loop evaluation pipelines. Off-the-shelf open-loop imitation works best where environmental guardrails are feasible.

Waymo’s system starts with a large, off-board foundation model understanding the physical world. This is specialized into three 'teacher' models: the Driver, the Simulator, and the Critic. These teachers then train smaller, efficient 'student' models that run in the vehicle.

Beyond supervised fine-tuning (SFT) and human feedback (RLHF), reinforcement learning (RL) in simulated environments is the next evolution. These "playgrounds" teach models to handle messy, multi-step, real-world tasks where current models often fail catastrophically.

A pure 'pixels in, actions out' model is insufficient for full autonomy. Waymo augments its end-to-end learning with structured, intermediate representations (like objects and road concepts). This provides crucial knobs for scalable simulation, safety validation, and defining reward functions.

A pure "pixels-in, actions-out" model is insufficient for full autonomy. While easy to start, this approach is extremely inefficient to simulate and validate for safety-critical edge cases. Waymo augments its end-to-end system with intermediate representations (like objects and road signs) to make simulation and validation tractable.

A "vanilla" end-to-end model is insufficient for safety-critical systems. Waymo's foundation model is end-to-end but is augmented with "structured materialized intermediate representation." This allows for crucial runtime validation, richer training, and closed-loop evaluation necessary for superhuman performance at scale.

Waymo uses a foundation model to create specialized, high-capacity "teacher" models (Driver, Simulator, Critic) offline. These teachers then distill their knowledge into smaller, efficient "student" models that can run in real-time on the vehicle, balancing massive computational power with on-device constraints.

The core technology for self-driving has evolved. Early systems relied on 'imitation learning' (copying human drivers). The current state-of-the-art involves end-to-end reinforcement learning within a closed-loop simulation, allowing the system to learn optimal behaviors on its own, far surpassing simple imitation.