Early AI adoption metrics focused on usage (e.g., tokens consumed), leading to wasteful “token maxing.” Successful teams quickly pivoted to measuring real business impact, such as the overall speed of the software delivery lifecycle, to gauge AI effectiveness and drive true ROI.

Instead of measuring how many questions a chatbot answered, the bank tracked the reduction in support tickets for the corresponding human team, which dropped 60%. This focus on tangible business outcomes provides a much clearer picture of AI's real value than simple activity metrics.

The company initially tracked vanity metrics like message counts and tokens used. They quickly pivoted to measuring AI's success by its tangible business impact, such as increased partner-facing time and the number of workflows automated, avoiding the trap of rewarding mere activity.

While many teams track token usage as a primary AI ROI metric, Anthropic views it differently. High token usage doesn't correlate with high value, as it's easy to waste tokens. Instead, a token count of zero is the most important signal, as it clearly indicates someone is not using the AI at all.

Traditional product metrics like DAU are meaningless for autonomous AI agents that operate without user interaction. Product teams must redefine success by focusing on tangible business outcomes. Instead of tracking agent usage, measure "support tickets automatically closed" or "workflows completed."

According to Mike Cannon-Brookes, advanced enterprises are not tracking AI success by counting tokens. Instead, they are asking harder questions about overall output, such as engineering productivity and quality. They understand that high token usage doesn't always correlate with high productivity, shifting focus from raw usage to tangible business outcomes.

Early AI adoption metrics focused on usage, like tokens consumed by engineers ('token maxing'). This incentivized wasteful activity. Mature organizations now measure AI's impact on core business metrics, such as the speed of shipping code from idea to production, which provides a true measure of value.

Open and click rates are ineffective for measuring AI-driven, two-way conversations. Instead, leaders should adopt new KPIs: outcome metrics (e.g., meetings booked), conversational quality (tracking an agent's 'I don't know' rate to measure trust), and, ultimately, customer lifetime value.

To gauge real AI integration, ignore official strategies and look at the data. Low per-employee token spend is a red flag indicating a lack of genuine, hands-on usage and curiosity, which are the real drivers of successful adoption.