Even within OpenAI, a stark performance gap exists. Engineers who avoid using agentic AI for coding are reportedly 10x less productive across metrics like code volume, commits, and business impact. This creates significant challenges for performance management and HR.

METR's influential study on AI developer productivity is now difficult to replicate. As AI tools become more powerful, developers are unwilling to be randomized into a control group where AI use is forbidden. This selection bias makes it increasingly impractical to measure true productivity gains with the original study design.

There's a significant gap between AI performance on structured benchmarks and its real-world utility. A randomized controlled trial (RCT) found that open-source software developers were actually slowed down by 20% when using AI assistants, despite being miscalibrated to believe the tools were helping. This highlights the limitations of current evaluation methods.

A 2025 study revealed a stark gap between developers' perceived AI-driven productivity gains and their actual, measured performance. This suggests the feeling of speed from using AI tools is a powerful, but potentially misleading, metric for true effectiveness.

A randomized controlled trial revealed a nearly 40% perception gap in developer productivity. While experienced developers using AI tools were measurably 19% slower, they self-reported feeling 20% faster. This highlights the unreliability of self-reported metrics for assessing AI's impact.

Human intuition is a poor gauge of AI's actual productivity benefits. A study found developers felt significantly sped up by AI coding tools even when objective measurements showed no speed increase. The real value may come from enabling tasks that otherwise wouldn't be attempted, rather than simply accelerating existing workflows.

A randomized controlled trial by AI evaluation non-profit METR showed experienced open-source developers were 19% slower when using AI tools. This contradicts the common narrative and the developers' own perception that they were 20% faster, highlighting a significant gap between perceived and measured productivity.

While companies report low official adoption, about 50% of workers use AI and hide the resulting productivity gains. This 'shadow adoption' stems from fear that revealing AI's efficiency will lead to layoffs instead of rewards, preventing companies from capitalizing on the technology's full potential.

A recent study found that AI assistants actually slowed down programmers working on complex codebases. More importantly, the programmers mistakenly believed the AI was speeding them up. This suggests a general human bias towards overestimating AI's current effectiveness, which could lead to flawed projections about future progress.