Skip to content

Knowledge base

AI productivity studies: METR, Stanford, DORA and Multitudes

No single AI productivity number holds. METR measured experienced developers 19% slower in repositories they knew, Stanford found a median gain near 10% that shrinks as tasks and codebases grow, DORA finds AI amplifies the existing system, and Multitudes ties gains to learning time.

Written by

Updated:

Evidence 7

Our position. We hold that usage is not ROI. A team can use the tool heavily and ship no faster, because the constraint was never the typing, so we measure delivered outcome rather than tool usage (Usage is not ROI (opens in a new tab)).

Four studies support that position from different angles, because each counts something different. METR timed task completion in a controlled trial, Stanford measured output across a large developer dataset, DORA surveyed teams, and Multitudes combined surveys, interviews and telemetry.

Did AI make experienced developers faster in METR’s trial?

METR ran a randomized controlled trial with 16 experienced open-source developers working in repositories they knew well. With early-2025 AI tools, participants took 19% longer to complete their tasks. They still expected the tools to make them faster. METR describes the result as a snapshot of early-2025 capabilities in one relevant setting, not a general law about every developer or task (METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)).

The study makes two points useful for measurement. Perceived speed can diverge from measured completion time, and realistic work in a familiar codebase can produce a different result from self-contained coding benchmarks.

Where do AI gains show up in Stanford’s data?

Stanford research presented across two public talks reports a median productivity gain around 10% across a large developer dataset. The reported gains fall as task complexity and codebase size rise, while codebase cleanliness correlates more strongly with productivity than token use (Stanford software engineering productivity research talk (opens in a new tab)).

A four-month enterprise comparison shows why output needs quality guardrails. Pull requests increased while code quality fell and rework rose, leaving effective output broadly flat (Stanford, measuring AI ROI talk (opens in a new tab)). These results support measuring delivery and rework together rather than treating generated code or pull-request counts as productivity.

What does DORA mean by AI as an amplifier?

DORA’s 2025 report describes AI as an amplifier of existing organizational strengths and weaknesses. The report recommends improving the underlying system and measuring software delivery outcomes instead of adoption alone (DORA, State of AI-assisted Software Development 2025 (opens in a new tab)).

This makes maturity a condition, not a guarantee. Faster generation can expose review, testing, and deployment constraints that were already present.

What does AI adoption depend on, according to Multitudes?

Multitudes combined survey responses, interviews, and telemetry in its research on how engineering organizations adapt to AI. Its public findings emphasize time to learn, peer support, and changes to the codebase and development process. The research also shows a gap between desired outcomes and the measures organizations use (Multitudes research (opens in a new tab)).

The evidence supports treating adoption as an organizational change with explicit learning time. A license or usage target does not establish productivity.

How do you read the studies together?

  • Compare like with like: task completion, delivery throughput, quality, and self-reported time savings answer different questions.
  • Pair output measures with rework, review time, defects, and delivery stability.
  • State the population, task type, and measurement window beside every number.
  • Treat vendor and practitioner reports as directional evidence unless their methods and data are public.

Resources