Skip to content

Knowledge base

Productivity evidence and quality

Measured productivity gains from AI coding are modest and conditional, and one randomized trial found a slowdown: they vary by task, codebase and enablement, self-reports overstate them, and higher output raises churn, defects and review load unless quality gates scale with it.

Written by

Updated:

Getting Started 2

Our position. Productivity is not value (opens in a new tab): AI inflates what is easy to measure, such as code volume and pull request count, and is neutral on value.

What should you take from the evidence?

  • Check a study’s task mix before comparing it: greenfield low-complexity work gains most, brownfield high-complexity work least.
  • Check how much enablement participants had: METR’s started with a 30-minute introduction and onboarding call.
  • Read vendor claims as marketing until a controlled study or production telemetry backs them.

How much faster do developers get with AI?

Rigorous studies find gains that are modest and conditional, and one randomized trial found a slowdown. The METR randomized study found 16 experienced open-source developers were ~19% slower with AI on their own repositories, having forecast a ~24% speedup beforehand and still estimating ~20% afterwards (METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)). A rarely cited methodological caveat is that participants started with a 30-minute introduction and onboarding call that included basic Cursor training, and METR cannot rule out learning effects beyond 50 hours of Cursor use (METR paper, appendix G.3 (opens in a new tab); METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)). Post-hoc analyses add texture: only 39% of Cursor generations were accepted and developers often reworked even accepted code, screen recordings show notably more idle time with AI (waiting for tools breaks flow), and the robustness checks hold up, with 84% of AI-allowed recordings showing actual AI use (Second Thoughts - Not So Fast: AI Coding Tools Can Actually Reduce Productivity (opens in a new tab)).

The one participant with more than 50 hours of prior Cursor experience did see a positive speedup, though 56% of participants had never used Cursor before and a study author explicitly rejects the learning curve as a sufficient explanation, listing at least five likely contributing factors (Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)).

The Stanford study of 120,000+ developers found a ~10% median gain, and an environment cleanliness index of the codebase correlating with the gains (R-squared 0.40, “a pretty decent correlation” in the talk’s words, versus a loose 0.20 for token usage). The Stanford group’s talks put a gains matrix behind that median: greenfield low-complexity work sees 30-40% gains, greenfield high-complexity 10-15%, brownfield low-complexity 15-20%, and brownfield high-complexity 0-10%; gains decay sharply as codebase size grows and can turn negative in less popular languages like COBOL, Haskell, or Elixir. Self-assessment misjudges by roughly 30 percentile points, and the same telemetry found roughly 10% of engineers to be “ghost engineers” with essentially no detectable output (Stanford software engineering productivity research talk (opens in a new tab)).

A MIT/Microsoft/Princeton study across 4,867 developers measured a 26% increase in completed tasks per week, far from the 10x marketing claims but substantial (From Trees to Forest: The Real Impact of AI on Developer Productivity (opens in a new tab)).

Perception and telemetry diverge systematically. DX’s Q1 2026 benchmark across 400+ companies reports 3.9 hours of self-reported weekly time savings and 27.4% AI-authored code, yet its own longitudinal study (Nov 2024 to Feb 2026) shows PR throughput gains of only 10-15% over more than a year, substantially lower than the self-reports imply (DX Q1 2026 AI-Assisted Engineering Impact Report). Survey design inflates numbers further: Atlassian’s 2025 State of Developer Experience survey of 3,500 engineers asked about time saved across the entire working week rather than coding alone, and got 99% of developers reporting more than four hours saved weekly, 46% reporting 11 to 20 hours; the same population reports losing more than 10 hours a week to organizational inefficiencies, and 63% say their leaders do not understand the challenges they face (Atlassian - State of Developer Experience 2025 talk (opens in a new tab)).

The METR participants illustrate the same effect at the individual level: they expected AI to make them faster while measured completion time moved in the opposite direction (METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)). Any claim built purely on self-reported time savings should be discounted accordingly.

Averages also hide variance between organizations and individuals. DX presented company-level quality metrics ranging from improvement to deterioration, while DORA 2025 describes AI as an amplifier of the surrounding engineering system (DX, AI strategy playbook talk (opens in a new tab); DORA, State of AI-assisted Software Development 2025 (opens in a new tab)). The Q1 2026 DX data adds segment detail: industry adoption reached 93%, engineering managers who use AI daily ship 4x as much code as six months earlier, junior engineers overtook Staff+ in weekly time saved (4.9 versus 4.8 hours), and companies with fewer than 200 developers outpace large enterprises (DX – AI-assisted engineering impact report, Q1 2026 (opens in a new tab), vendor benchmark).

A year-long production deployment at 300 engineers measured a 31.8% reduction in PR review cycle time with ~40% of shipped code AI-generated (Production study - Intuition to Evidence: Measuring AI’s True Impact on Developer Productivity (opens in a new tab)). That deployment, Tata 1mg’s in-house DeputyDev with six parallel review agents on Claude Sonnet, also shows amplification within one company: the top 30 adopters shipped 61.3% more code while the bottom 30 shipped 11.4% less, a 72.7-percentage-point gap, with junior engineers gaining the most at +77%; automatic acceptance of AI suggestions caused quality issues and was quickly disabled (Production study - Intuition to Evidence: Measuring AI’s True Impact on Developer Productivity (opens in a new tab)).

Intercom set an explicit goal of doubling engineering throughput, measured as code changes per R&D person, standardized on a single agent platform with a dedicated enablement team, and reached the doubling in under a year (Intercom talk - Doubling engineering throughput (opens in a new tab)). The gap between benchmark and reality remains the standing caution: models that solve 86% of LeetCode problems complete only 26% of real freelance tasks, which is why context, not model capability, is the binding constraint (How Engineering Teams use AI (opens in a new tab), March 2025).

What does faster output cost in quality?

The largest telemetry-based counterweight comes from Faros AI’s engineering report (April 2026), drawing on two years of data across 22,000 developers and 4,000+ teams. Acceptance of AI-generated code rose from 20% to 60% and output climbed (epics per developer +66%, task throughput +33.7%, PR merge rate +16.2%), but the downstream bill is steep: code churn up 861% under high adoption, the incidents-to-PR ratio up 242.7%, bugs per developer up 54%, PRs merged without any review up 31.3%, and median time in review up 441.5% as senior engineers absorb the review load. On telemetry grounds Faros disputes the DORA 2025 maturity thesis, finding high-performing organizations experiencing the same downstream deterioration as everyone else (Faros AI – The AI Engineering Report 2026: The AI Acceleration Whiplash (opens in a new tab), April 2026; Faros sells the telemetry platform surfacing these metrics).

Independent numbers point the same direction. A Stanford analysis found 40% of AI-generated code contains security vulnerabilities; Stack Overflow’s 2025 survey has 46% of developers distrusting AI accuracy (up from 31% in 2023), with 66% naming “almost right, but not quite” code as the biggest time sink; and Answer.AI found the autonomous agent Devin completed only 3 of 20 assigned tasks (From Trees to Forest: The Real Impact of AI on Developer Productivity (opens in a new tab), August 2025). Vendor claims sit far above these numbers, Cursor promising “10x developer productivity” at a $9B valuation, Replit claiming 90% of foundational code, Amazon citing 4,500 years of development time saved, and should be read as marketing (From Trees to Forest: The Real Impact of AI on Developer Productivity (opens in a new tab)).

Quality erosion can flip ROI negative outright. In a 350-person enterprise team the Stanford group measured four months before and after AI adoption: PRs rose 14%, but code quality fell 9% against industry benchmark, rework went up 2.5x, and effective output did not meaningfully change, leaving real license and token spend against flat productivity (Stanford - measuring AI ROI talk (opens in a new tab)). Their conclusion matches the consensus on Impact measurement: pick one primary output metric and surround it with quality guardrail metrics.

Resources

Articles

Papers and reports

Posts

Talks and videos