A randomized controlled trial by AI evaluation non-profit METR showed experienced open-source developers were 19% slower when using AI tools. This contradicts the common narrative and the developers' own perception that they were 20% faster, highlighting a significant gap between perceived and measured productivity.
When AI research firm METR tried to repeat its productivity study, 30-50% of developers declined to participate because they didn't want to forgo AI access. This selection bias makes establishing a true baseline for comparison nearly impossible, suggesting that measuring AI's true impact is becoming methodologically unfeasible as adoption grows.
An analysis by Faro's AI found AI-assisted teams merged 98% more PRs, not by completing individual tasks faster, but by enabling developers to parallelize their workflow. Developers can kick off a task with an agent while simultaneously reviewing another human's work. This shows teams should optimize for throughput, not single-task velocity.
An AI agent's verification loop only confirms its code satisfies the existing test suite; it does not validate that the overall approach was correct. An agent can pass every test while implementing a flawed architecture or introducing a vulnerability nobody thought to write a test for, widening the gap between a 'green checkmark' and human approval.
A novel vulnerability arises when an AI agent references a package name that doesn't exist. Malicious actors can register these hallucinated package names and upload malicious code. This creates a documented supply chain attack vector that requires specific checks beyond typical static analysis to mitigate.
