Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Major AI breakthroughs like Transformers accelerate initial progress but are not silver bullets for the safety-critical long tail. The nature of the problem is that getting a prototype working is relatively easy, but achieving the final "nines" of reliability is incredibly difficult, justifying Google's early, multi-decade investment.

Related Insights

Tesla's camera-only system gives it a significant cost advantage over Waymo's LiDAR-equipped vehicles. However, current data shows a Waymo vehicle crashes every 400,000 miles, while Tesla's crashes every 50,000. Tesla's ability to scale hinges entirely on proving its cheaper technology can become as safe.

Unlike traditional engineering, breakthroughs in foundational AI research often feel binary. A model can be completely broken until a handful of key insights are discovered, at which point it suddenly works. This "all or nothing" dynamic makes it impossible to predict timelines, as you don't know if a solution is a week or two years away.

Rivian's CEO explains that early autonomous systems, which were based on rigid rules-based "planners," have been superseded by end-to-end AI. This new approach uses a large "foundation model for driving" that can improve continuously with more data, breaking through the performance plateau of the older method.

A pure "pixels-in, actions-out" model is insufficient for full autonomy. While easy to start, this approach is extremely inefficient to simulate and validate for safety-critical edge cases. Waymo augments its end-to-end system with intermediate representations (like objects and road signs) to make simulation and validation tractable.

Drawing from his Tesla experience, Karpathy warns of a massive "demo-to-product gap" in AI. Getting a demo to work 90% of the time is easy. But achieving the reliability needed for a real product is a "march of nines," where each additional 9 of accuracy requires a constant, enormous effort, explaining long development timelines.

While initial safety validation is crucial, the bigger, long-term problem is ensuring safety across thousands of vehicles over many years. This involves managing part obsolescence, configuration drift, and real-time performance monitoring to prevent a fleet-wide grounding event, similar to challenges in the airline industry.

Dropbox's AI strategy is informed by the 'march of nines' concept from self-driving cars, where each step up in reliability (90% to 99% to 99.9%) requires immense effort. This suggests that creating commercially viable, trustworthy AI agents is less about achieving AGI and more about the grueling engineering work to ensure near-perfect reliability for enterprise tasks.

Despite rapid software advances like deep learning, the deployment of self-driving cars was a 20-year process because it had to integrate with the mature automotive industry's supply chains, infrastructure, and business models. This serves as a reminder that AI's real-world impact is often constrained by the readiness of the sectors it aims to disrupt.

Achieving near-perfect AV reliability (99.999%) is exponentially harder than getting to 99%. This final push involves solving countless subtle, city-specific issues, from differing traffic light colors and curb heights to unique local sounds like emergency sirens, which vehicles must recognize.

The public holds new technologies to a much higher safety standard than human performance. Waymo could deploy cars that are statistically safer than human drivers, but society would not accept them killing tens of thousands of people annually, even if it's an improvement. This demonstrates the need for near-perfection in high-stakes tech launches.

AV Progress Is Deceptively Easy Initially; Long-Term Investment Is Key for Safety | RiffOn