Developers claiming 10x speedups from AI often aren't 10x faster on their core tasks. Instead, they're tackling new side projects that were previously impossible, creating a perception of "infinite" speedup. However, these new tasks are often less economically valuable, inflating the true productivity gain on business-critical work.

There's a significant gap between AI performance on structured benchmarks and its real-world utility. A randomized controlled trial (RCT) found that open-source software developers were actually slowed down by 20% when using AI assistants, despite being miscalibrated to believe the tools were helping. This highlights the limitations of current evaluation methods.

A 2025 study revealed a stark gap between developers' perceived AI-driven productivity gains and their actual, measured performance. This suggests the feeling of speed from using AI tools is a powerful, but potentially misleading, metric for true effectiveness.

When AI research firm METR tried to repeat its productivity study, 30-50% of developers declined to participate because they didn't want to forgo AI access. This selection bias makes establishing a true baseline for comparison nearly impossible, suggesting that measuring AI's true impact is becoming methodologically unfeasible as adoption grows.

A randomized controlled trial revealed a nearly 40% perception gap in developer productivity. While experienced developers using AI tools were measurably 19% slower, they self-reported feeling 20% faster. This highlights the unreliability of self-reported metrics for assessing AI's impact.

Human intuition is a poor gauge of AI's actual productivity benefits. A study found developers felt significantly sped up by AI coding tools even when objective measurements showed no speed increase. The real value may come from enabling tasks that otherwise wouldn't be attempted, rather than simply accelerating existing workflows.

A recent study found that AI assistants actually slowed down programmers working on complex codebases. More importantly, the programmers mistakenly believed the AI was speeding them up. This suggests a general human bias towards overestimating AI's current effectiveness, which could lead to flawed projections about future progress.

While AI coding assistants appear to boost output, they introduce a "rework tax." A Stanford study found AI-generated code leads to significant downstream refactoring. A team might ship 40% more code, but if half of that increase is just fixing last week's AI-generated "slop," the real productivity gain is much lower than headlines suggest.

A Meta study found expert programmers were less productive with AI tools. The speaker suggests this is because users thought they were faster while actually being distracted (e.g., social media) waiting for the AI, highlighting a dangerous gap between perceived and actual productivity.