Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

While many teams track token usage as a primary AI ROI metric, Anthropic views it differently. High token usage doesn't correlate with high value, as it's easy to waste tokens. Instead, a token count of zero is the most important signal, as it clearly indicates someone is not using the AI at all.

Related Insights

The key measure of leverage for AI-powered developers is no longer GPU utilization (FLOPs) but the volume of tokens processed by agents. Karpathy feels nervous when his token subscriptions are underutilized, indicating he's the bottleneck, not the system.

When companies give employees AI token budgets and track usage on dashboards, it incentivizes ROI-negative behavior. Employees feel compelled to spend their entire allocation to appear productive, a classic example of Goodhart's Law where the metric (usage) undermines the goal (productivity).

A 'value premium' is emerging where users' reported value from AI grows faster than their usage time. Even users with flat usage hours report increasing value, demonstrating that skill development and learning curve payoffs are key drivers of AI ROI, independent of raw hours spent.

To get teams experimenting with AI, leaders should provide an open budget for tokens initially. Being 'profligate' at the start is crucial, as imposing constraints too early leads to unimpressive results, stifles creativity, and hinders true adoption. Efficiency can be optimized later.

Anthropic intentionally avoids using "user minutes" as a core metric. This strategic choice reflects their focus on safety and user well-being, aiming to build a helpful tool rather than an addictive product. By prioritizing value creation over engagement time, they steer clear of the incentive structures that can lead to psychologically harmful AI behaviors.

When companies measure AI adoption by counting tokens used, it creates a perverse incentive. Employees and their teams create agents to perform pointless tasks simply to boost their metrics, leading to fake productivity and problematic artifacts.

Sendbird tracks and ranks every employee's daily AI token usage on a public dashboard, categorizing them from 'AI Newbie' to 'AI God.' This gamified metric makes AI adoption a visible, shared company objective and identifies who needs enablement.

According to Mike Cannon-Brookes, advanced enterprises are not tracking AI success by counting tokens. Instead, they are asking harder questions about overall output, such as engineering productivity and quality. They understand that high token usage doesn't always correlate with high productivity, shifting focus from raw usage to tangible business outcomes.

High token consumption is framed as a key metric for AI leverage, not a cost. This goal forces teams to find ways to delegate more complex, long-running, and parallel tasks to AI agents, thus maximizing the intelligence and autonomous work extracted from the models.

Giving teams a 'token budget' is flawed because it incentivizes generating low-value output to hit a quota, similar to bad hiring quotas. Instead, companies must tie token consumption directly to business KPIs. This reframes AI spend as a value-creating investment, not a cost to be managed.

Use Token Usage to Identify AI Non-Adopters, Not Power Users | RiffOn