Early AI adoption metrics focused on usage (e.g., tokens consumed), leading to wasteful “token maxing.” Successful teams quickly pivoted to measuring real business impact, such as the overall speed of the software delivery lifecycle, to gauge AI effectiveness and drive true ROI.

The success of AI in marketing should not be measured by the quantity of content or ideas generated, which can create chaos. Instead, leaders must track its impact on core business metrics like revenue growth and operational efficiency. The goal is enabling a 10-person team to operate with the impact of a 100-person team.

The company initially tracked vanity metrics like message counts and tokens used. They quickly pivoted to measuring AI's success by its tangible business impact, such as increased partner-facing time and the number of workflows automated, avoiding the trap of rewarding mere activity.

DBS quantifies AI impact not by cost savings, but by the incremental revenue generated from AI-driven customer "nudges." Using rigorous A/B testing, they track the lift from these interactions, reframing AI's value proposition from an efficiency tool to a revenue growth engine, targeting over a billion dollars.

With infinitely scalable AI agents, cost and time per interaction are no longer primary constraints. Companies should abandon classic efficiency metrics like Average Handle Time and instead measure success by outcomes, such as percentage of tasks completed and improvements in Customer Satisfaction (CSAT).

Demanding a direct, line-item ROI for foundational AI initiatives is like asking for the ROI on Wi-Fi—it's the wrong question. Instead of getting bogged down in impossible calculations, leaders should focus on measuring the business outcomes enabled by the technology, such as innovation speed or new product creation. Obsess on outcomes, not direct financial return.

Traditional product metrics like DAU are meaningless for autonomous AI agents that operate without user interaction. Product teams must redefine success by focusing on tangible business outcomes. Instead of tracking agent usage, measure "support tickets automatically closed" or "workflows completed."

According to Mike Cannon-Brookes, advanced enterprises are not tracking AI success by counting tokens. Instead, they are asking harder questions about overall output, such as engineering productivity and quality. They understand that high token usage doesn't always correlate with high productivity, shifting focus from raw usage to tangible business outcomes.

A common failure is defining an AI pilot's success with engineering metrics like accuracy or latency. True success is a business outcome, such as the finance team trusting the AI's output enough to stop manually double-checking it. Success metrics must be framed in terms a CFO would accept.