/
© 2026 RiffOn. All rights reserved.

Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

  1. The a16z Show
  2. Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone
Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show · Oct 18, 2026

EliseAI’s Mario Martone: production AI in housing/healthcare needs more than benchmarks—scale, latency, evaluations, hybrid ML.

A 90% AI benchmark is catastrophic at 60M calls

Mario Martone says a model that looks reliable at 90% in a paper would mean enormous failures across 60 million yearly calls. He notes five or six test conversations can look good, but at millions of interactions different methods diverge sharply, making evaluation at production scale essential.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago

Renewal predictions need neural nets, not LLMs

EliseAI predicts lease renewals with an in-house neural network, not an LLM, because it has lots of behavioral data and needs reliable probabilities. Martone says LLMs struggle with determinism and probability, so his team picks the tool based on impact and shipping speed rather than defaulting to LLMs.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago

Ignore public model benchmarks; test on your own calls

Martone says when a new model appears on Slack or X, he does not trust the public benchmark. His team sets up an internal experiment and compares every model on EliseAI's own benchmarks, because external scores do not reveal performance on real calls at scale.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago

Ten reviewers can monitor millions of AI conversations

EliseAI runs fully automated loops handling about one million calls a day plus hundreds of thousands of emails and texts. Martone says building evaluations lets roughly ten people understand daily how those housing and healthcare conversations are going, even when no one reviews each interaction.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago

Physics-style idea generation fuels open-source model fine-tuning

Martone brought theoretical physics' idea-generation process to EliseAI's applied research. He says his team spent months fine-tuning open-source models, and because researchers were used to tackling hard open questions, they trained their own models and put them into production with strong results, according to Martone.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago

Affordable housing is a huge, nearly unautomated AI opportunity

Martone says affordable housing is huge, complex, and almost untouched by technology, with nearly no tech depth in the industry. EliseAI is trying to cut the cost of running it because it affects the most vulnerable renters and is a massive part of real estate.

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone thumbnail

Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

The a16z Show·a month ago