To quantify the real-world impact of its AI tools, Block tracks a simple but powerful metric: "manual hours saved." This KPI combines qualitative and quantitative signals to provide a clear measure of ROI, with a target to save 25% of manual hours across the company.

Coinbase held a time-boxed event where 100+ engineers used an AI tool to simultaneously submit PRs for trivial fixes. This created a transformational moment, breaking inertia, proving the tool's value, and generating massive, visible momentum for adoption across the entire organization.

Superhuman adopted AI coding tools using a three-quarter plan: 1) Unrestricted experimentation with centralized budget approval. 2) Analysis and measurement using self-reported PR labels. 3) Observing a sustained increase in engineering throughput from 4 to 6 PRs per engineer per week.

Block's CTO quantifies the impact of their internal AI agent, Goose. AI-forward engineering teams save 8-10 hours weekly, a figure he considers the absolute baseline. He notes, "this is the worst it will ever be," suggesting exponential gains are coming.

Instead of focusing on headcount reduction, Goldman's CIO measures the success of developer AI tools by their ability to consistently help projects finish ahead of schedule. This provides a tangible metric for increased output and organizational capacity.

AI tools provide quantifiable productivity gains in technical fields. Developers using GitHub Copilot, for instance, finish tasks approximately 55% faster. Furthermore, 88% of these developers report feeling more productive, demonstrating that AI augmentation leads to significant and measurable improvements in workflow efficiency and employee satisfaction.

Measuring AI's impact by output metrics like 'percent of agent-written code' or 'number of PRs merged' is a trap. These metrics say nothing about value. Instead, focus on counterbalance metrics that measure quality and meaningful impact, such as a reduction in bugs or positive user feedback.

The latest AI coding assistants facilitate a massive leap in developer productivity. The host demonstrated this by merging 44 pull requests and adding nearly 93,000 lines of code in just five days, a workload that would typically take an entire team months to complete, making the scale of the impact concrete.

Contrary to fears that AI creates low-quality "slop," Intercom found their code quality improved. AI compresses the cost of fixing tech debt, flaky tests, and other internal projects, making it easier for the business to invest in them.