Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Josh Gao says it is expected that models like GPT-6 Astra generalize to robot tasks because robotics data and action trajectories are being added to pre-training mixes of large foundation models. He still sees room for specialized robot models that reach four-nines reliability in narrow tasks.

Related Insights

To build generalist robots, the most effective approach is pre-training foundation models on internet-scale video datasets, not just simulation or tele-operated data. This vast, diverse data provides a deep, implicit understanding of physics and object interaction that is impossible to replicate in controlled environments, enabling true generalization.

The Physical Intelligence thesis is that a foundation model learning from diverse data can achieve a "physical understanding" of the world, making it easier to adapt to new tasks than building single-purpose robots from scratch. Generality leverages broader data, which is ultimately a more scalable approach.

Building narrow AI creates high performance on one or two tasks, but the nth task costs as much to train as the first. In contrast, building generalist baselines endows models with broad common sense and physical reasoning. Once this broad base is established, tuning a robot to high reliability and mastery on any arbitrary task becomes significantly faster and cheaper.

Figure is observing that data from one robot performing a task (e.g., moving packages in a warehouse) improves the performance of other robots on completely different tasks (e.g., folding laundry at home). This powerful transfer learning, enabled by deep learning, is a key driver for scaling general-purpose capabilities.

Language models execute identically regardless of device, but robotics software is constrained by the physical robot body. Keerthana Gopalakrishnan places current robotics in the 'GPT-2 era' because generic robotic brains cannot reliably control arbitrary, unseen robot embodiments out of the box, nor can they reliably generalize from few-shot demonstrations across hard, diverse tasks.

Adopting the true 'foundation model ethos' from LLMs is difficult for roboticists. It means a warehouse automation company should collect data from kitchen robots. This breadth, while seemingly unrelated, builds a generalist model that better handles the weird edge cases in the target domain than a narrowly trained specialist model.

Physical Intelligence demonstrated an emergent capability where its robotics model, after reaching a certain performance threshold, significantly improved by training on egocentric human video. This solves a major bottleneck by leveraging vast, existing video datasets instead of expensive, limited teleoperated data.

Runway’s robotics thesis is that pre-training on massive, easily available third-person video data (e.g., people performing tasks) is more scalable and effective than relying on expensive, limited teleoperation or first-person data. This general world knowledge can then be fine-tuned for specific robotic tasks.

The adoption of powerful AI architectures like transformers in robotics was bottlenecked by data quality, not algorithmic invention. Only after data collection methods improved to capture more dexterous, high-fidelity human actions did these advanced models become effective, reversing the typical 'algorithm-first' narrative of AI progress.

Josh Gao says scaling the current data recipe by orders of magnitude will solve many constrained tasks. But a drop-in robot that handles any task amid real-world complexity, like weather, is still far off; he expects tactile, simulation, and even smell-like modalities to enter omni-model training.