Early AI adoption metrics focused on usage (e.g., tokens consumed), leading to wasteful “token maxing.” Successful teams quickly pivoted to measuring real business impact, such as the overall speed of the software delivery lifecycle, to gauge AI effectiveness and drive true ROI.

Enterprises that track and reward high AI token usage risk incentivizing the wrong behavior. This is a modern "Cobra Effect," where employees generate unnecessary output to hit metrics, much like people who bred cobras to collect a bounty. The focus must be on utility, not volume.

To combat engineer skepticism, companies incentivized AI usage to the point of wastefulness ("token maxing"). This overcorrection is a faster way to achieve broad adoption than starting with strict controls. You can enforce responsible usage and cost efficiency once the value is proven and ingrained.

The company initially tracked vanity metrics like message counts and tokens used. They quickly pivoted to measuring AI's success by its tangible business impact, such as increased partner-facing time and the number of workflows automated, avoiding the trap of rewarding mere activity.

When companies measure AI adoption by counting tokens used, it creates a perverse incentive. Employees and their teams create agents to perform pointless tasks simply to boost their metrics, leading to fake productivity and problematic artifacts.

True ROI of AI isn't found in usage metrics like token counts. It's measured by identifying entire, expensive projects (e.g., a $4M manual document conversion) and using AI to make the problem 'vanish,' completing the work in hours instead of months.

Superhuman's CEO advises against simply tracking AI costs, a practice he calls 'token maxing'. Instead, they evaluate the ROI of internal AI tools by measuring developer productivity metrics like feature delivery pace. This output-focused approach has doubled engineering velocity, justifying the AI spend.

According to Mike Cannon-Brookes, advanced enterprises are not tracking AI success by counting tokens. Instead, they are asking harder questions about overall output, such as engineering productivity and quality. They understand that high token usage doesn't always correlate with high productivity, shifting focus from raw usage to tangible business outcomes.

Don't rely on traditional project milestones to gauge AI progress. Instead, measure success through granular unit economics and operational metrics. Metrics like 'cost per release' or 'cycle time per feature' provide immediate feedback on whether your strategic hypothesis is valid, enabling rapid iteration.