Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Josh Gao says the industry is converging on roughly a million hours of human egocentric data, tens of thousands of hours of UMI data, and a few hours of last-mile teleoperation or in-domain demonstrations. He stresses that tactile and simulation data have not even entered the mix yet.

Related Insights

Companies like One X deploy robots that are remotely operated by humans to complete tasks. This strategy provides immediate value to customers while simultaneously collecting vast amounts of real-world training data, which is the primary bottleneck for developing full autonomy.

Pre-training on massive human egocentric video gives foundation models a broad prior across diverse physical manipulation tasks. Fine-tuning with small batches of robot-specific data bridges embodiment gaps and establishes low-level dexterity without expensive massive teleoperation collection.

Robotics company OneX designs its robot hands to be biomechanically identical to human hands not for aesthetics, but for data transfer. This allows them to train models on vast amounts of existing human video, which then 'just works' on the robot, bypassing the need for extensive simulation or teleoperation data.

Physical Intelligence demonstrated an emergent capability where its robotics model, after reaching a certain performance threshold, significantly improved by training on egocentric human video. This solves a major bottleneck by leveraging vast, existing video datasets instead of expensive, limited teleoperated data.

Runway’s robotics thesis is that pre-training on massive, easily available third-person video data (e.g., people performing tasks) is more scalable and effective than relying on expensive, limited teleoperation or first-person data. This general world knowledge can then be fine-tuned for specific robotic tasks.

While Figure's CEO criticizes competitors for using human operators in robot videos, this 'wizard of oz' technique is a critical data-gathering and development stage. Just as early Waymo cars had human operators, teleoperation is how companies collect the training data needed for true autonomy.

No single data collection method will solve robotics. Teleoperation provides high-precision ground truth for specific robots but scales poorly and degrades when hardware updates. Universal Manipulation Interface (UMI) data offers better scale and precision via sensors but requires hardware maintenance. Egocentric human video provides massive scale but lacks precise end-effector force signals and suffers from human-to-robot physical embodiment mismatch.

Unlike LLMs trained on vast digital text, humanoid robots need immense amounts of real-world physical data to learn simple tasks. It's estimated that 100 million hours—over 11,000 years' worth—of interaction data is needed to create truly smart, useful humanoids, highlighting the scale of the challenge.

China is accelerating robot training by collecting two data types. It is building expensive, centralized "robot training centers" for high-quality machine data. Simultaneously, firms like JD.com are crowdsourcing "egocentric data" by equipping hundreds of thousands of workers with sensor gear to record their physical movements at scale.

Josh Gao says scaling the current data recipe by orders of magnitude will solve many constrained tasks. But a drop-in robot that handles any task amid real-world complexity, like weather, is still far off; he expects tactile, simulation, and even smell-like modalities to enter omni-model training.

Robot training mix converges on human, UMI, and teleop data | RiffOn