We scan new podcasts and send you the top 5 insights daily.
Robin Ross says measurement still has to start with a business outcome, but it is unclear whether a human visit or an agent visit is the stronger signal. She floats the idea that one human visit could equal five agent visits, or that the sequence of agent then human matters.
Instead of measuring how many questions a chatbot answered, the bank tracked the reduction in support tickets for the corresponding human team, which dropped 60%. This focus on tangible business outcomes provides a much clearer picture of AI's real value than simple activity metrics.
A robust framework for measuring an AI agent's success requires a tiered approach. First, establish baseline quality (is it working correctly?). Then, measure user engagement (adoption, retention). Finally, connect these to top-line business impact (revenue, savings).
Focusing on machine-level performance indicators like tokens burned or model latency measures vendor revenue rather than genuine customer utility. Strategic success metrics must center on whether target human users or agents exhibit measurable, intended behavioral shifts.
With infinitely scalable AI agents, cost and time per interaction are no longer primary constraints. Companies should abandon classic efficiency metrics like Average Handle Time and instead measure success by outcomes, such as percentage of tasks completed and improvements in Customer Satisfaction (CSAT).
Traditional product metrics like DAU are meaningless for autonomous AI agents that operate without user interaction. Product teams must redefine success by focusing on tangible business outcomes. Instead of tracking agent usage, measure "support tickets automatically closed" or "workflows completed."
Traditional efficiency metrics like handle time are insufficient. To become a strategic asset, contact centers should adopt outcome-based metrics like a "Value Enhancement Score." This measures an agent's ability to not just solve problems but also deepen connections and convert new growth opportunities.
Open and click rates are ineffective for measuring AI-driven, two-way conversations. Instead, leaders should adopt new KPIs: outcome metrics (e.g., meetings booked), conversational quality (tracking an agent's 'I don't know' rate to measure trust), and, ultimately, customer lifetime value.
Pitié warns that an AI crawler is not a customer and a referral is not an agent. Different questions need different answers, so he recommends separate KPIs: content access for crawlers, human conversions for referrals, task completion for agents, and help from a brand's own chatbot.
As AI agents increasingly interact with software to perform tasks, a new field of "Agent Experience" (AX) is emerging. The same principles of identifying and resolving friction in human user journeys (UX) will need to be applied to optimize the performance and efficiency of these automated agents.
Just as businesses use Google Analytics to optimize for human conversion, a new discipline of "agent analytics" is required. This involves tracking which agents visit, what data they request, where their queries fail, and why they "bounce." Optimizing this agent journey will become as critical as traditional conversion rate optimization (CRO).