Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Sergey Levine explains that human teleoperation fixes mistakes during robot deployment, teaching the system how to recover after dropping an object. However, this only yields clumsy autonomy; reinforcement learning must then be introduced on top of the robot's own experience to teach the system how to avoid dropping objects in the first place.

Related Insights

To avoid failure, launch AI agents with high human control and low agency, such as suggesting actions to an operator. As the agent proves reliable and you collect performance data, you can gradually increase its autonomy. This phased approach minimizes risk and builds user trust.

The ideal human-AI relationship is symbiotic. AI operates at superhuman scale, but when it errs, a human corrects it. Capturing these corrections provides the highest-quality, highest-information-gain data for fine-tuning the next version of the AI agent, creating a powerful self-improving system.

In robotics, purely imitating human actions is insufficient. A model trained this way doesn't learn how to recover from inevitable errors. Comma AI solves this by training its models in a simulator where they are forced to learn recovery paths from off-course situations, a critical step for real-world deployment.

While many in the robotics industry chase the "fully autonomous" narrative, teleoperation—having remote workers control machines with Xbox controllers—is an extremely valuable and practical step. Customers care about task completion, not the level of autonomy, making teleop a key tool for gathering training data and ensuring reliability.

Reinforcement Learning (RL) is ideal for fine-tuning AI agents because it allows them to self-learn and align their behavior by exploring vast, non-deterministic environments. This is a more scalable approach than supervised fine-tuning, which would require an impossibly large, pre-curated dataset to cover all potential pathways.

Many AI projects fail to reach production because of reliability issues. The vision for continual learning is to deploy agents that are 'good enough,' then use RL to correct behavior based on real-world errors, much like training a human. This solves the final-mile reliability problem and could unlock a vast market.

Beyond supervised fine-tuning (SFT) and human feedback (RLHF), reinforcement learning (RL) in simulated environments is the next evolution. These "playgrounds" teach models to handle messy, multi-step, real-world tasks where current models often fail catastrophically.

While Figure's CEO criticizes competitors for using human operators in robot videos, this 'wizard of oz' technique is a critical data-gathering and development stage. Just as early Waymo cars had human operators, teleoperation is how companies collect the training data needed for true autonomy.

According to Comma AI's CTO, the next frontier in robotics isn't just bigger models, but solving three fundamental challenges: 1) using ML for low-level controls, 2) making reinforcement learning (RL) practical for noisy environments, and 3) enabling continual, on-device learning to adapt to changing conditions.

For AI agents performing multi-step tasks, the ability to recognize, step back, and correct a mistake is arguably more critical for reliability than initial accuracy. While humans also make errors, our ability to backtrack is essential for completing complex, sequential objectives. This error-correction capability is a key feature of advanced reasoning models.