Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Josh Gao says wrist cameras are a hacky substitute for measuring contact force. Mecka is building gloves that capture how much force daily tasks require because no off-the-shelf wearable reliably records force and turns it into ready robot-training datasets.

Related Insights

Josh Gao says the industry is converging on roughly a million hours of human egocentric data, tens of thousands of hours of UMI data, and a few hours of last-mile teleoperation or in-domain demonstrations. He stresses that tactile and simulation data have not even entered the mix yet.

Pure visual feedback is insufficient for delicate real-world manipulation tasks such as stacking brittle items or assembling components. To avoid crushing materials, robots require real-time end-effector force sensing and physical back-drivability/compliance. Having explicit torque and pressure signals allows neural policies to make safe adjustments that vision alone cannot deduce.

To overcome the data bottleneck in robotics, Sunday developed gloves that capture human hand movements. This allows them to train their robot's manipulation skills without needing a physical robot for teleoperation. By separating data gathering (gloves) from execution (robot), they can scale their training dataset far more efficiently than competitors who rely on robot-in-the-loop data collection methods.

Instead of loading robots with costly sensors for touch or force, powerful learning models can infer physical properties from simple cameras. A wrist camera can act as a "touch sensor in disguise" by observing local deformations, dramatically lowering hardware costs and complexity for scalable robotics.

Instead of deploying thousands of expensive robots to gather manipulation data, Sunday Robotics is distributing cheaper, specialized gloves. This allows them to collect high-quality, diverse data from humans performing tasks in their own homes, accelerating model development.

Catheter assembly remains a surprisingly manual process because experienced operators develop a tactile sense for applying the right amount of force and making subtle adjustments. This 'touchy-feely' expertise, which varies from part to part, is difficult and often financially impractical to replicate with robotic automation.

According to Agility Robotics' co-founder, perception is now a largely solved problem. The new frontier is generating training data for robot control—the specific torque commands and sensor inputs for actions. Unlike text or images for LLMs, this data does not exist on the internet and must be painstakingly created.

No single data collection method will solve robotics. Teleoperation provides high-precision ground truth for specific robots but scales poorly and degrades when hardware updates. Universal Manipulation Interface (UMI) data offers better scale and precision via sensors but requires hardware maintenance. Egocentric human video provides massive scale but lacks precise end-effector force signals and suffers from human-to-robot physical embodiment mismatch.

Surgeons perform intricate tasks without tactile feedback, relying on visual cues of tissue deformation. This suggests robotics could achieve complex manipulation by advancing visual interpretation of physical interactions, bypassing the immense difficulty of creating and integrating artificial touch sensors.

Josh Gao says scaling the current data recipe by orders of magnitude will solve many constrained tasks. But a drop-in robot that handles any task amid real-world complexity, like weather, is still far off; he expects tactile, simulation, and even smell-like modalities to enter omni-model training.

Tactile force data needs custom gloves, not off-the-shelf sensors | RiffOn