Mario Martone says a model that looks reliable at 90% in a paper would mean enormous failures across 60 million yearly calls. He notes five or six test conversations can look good, but at millions of interactions different methods diverge sharply, making evaluation at production scale essential.
EliseAI predicts lease renewals with an in-house neural network, not an LLM, because it has lots of behavioral data and needs reliable probabilities. Martone says LLMs struggle with determinism and probability, so his team picks the tool based on impact and shipping speed rather than defaulting to LLMs.
Martone says when a new model appears on Slack or X, he does not trust the public benchmark. His team sets up an internal experiment and compares every model on EliseAI's own benchmarks, because external scores do not reveal performance on real calls at scale.
EliseAI runs fully automated loops handling about one million calls a day plus hundreds of thousands of emails and texts. Martone says building evaluations lets roughly ten people understand daily how those housing and healthcare conversations are going, even when no one reviews each interaction.
Martone brought theoretical physics' idea-generation process to EliseAI's applied research. He says his team spent months fine-tuning open-source models, and because researchers were used to tackling hard open questions, they trained their own models and put them into production with strong results, according to Martone.
Martone says affordable housing is huge, complex, and almost untouched by technology, with nearly no tech depth in the industry. EliseAI is trying to cut the cost of running it because it affects the most vulnerable renters and is a massive part of real estate.
