Measuring AI impact on engineering: what to count and what to ignore

AI pushes up every number that is easy to count. Code volume, pull requests, completed tasks and tool usage all rise, whether or not the customer ends up with anything more. Studies that measured outcomes put the typical gain at about 10 percent, and companies using the same tools land far apart.

So measure delivery rather than activity: what reaches production, how long it takes to get there, what breaks, and what had to be fixed. Take your own baseline before you copy someone else’s playbook. And expect things to dip before they improve: in our experience, AI first absorbs more capacity than it frees up.

The sections below follow the questions in the order engineering leaders usually ask them.

Why do the usual metrics mislead?

Because AI inflates precisely the things that are easy to measure:

• Code volume rises because generating code is cheap.
• PR count rises because smaller, faster changes feel productive.
• Task completion rises because people pick the easy tasks first.
• Story point velocity rises because points measure effort, not impact.

The dashboards go up, and leadership celebrates. The part that decides value goes unmeasured: features customers actually use, code that lasts six months without a rewrite, the time spent fixing AI output set against the time AI “saved”, technical debt per feature, and the AI rework ratio on generated code.

An illustration, not a measurement, makes the gap concrete. Picture a team that ships three times as much code while its users see 1.1 times as many useful features. The missing 1.9 went into code that needed fixing at once, into features no one asked for, and into complexity that drags on everything else. AI is good at productivity and neutral on value.

Usage numbers suffer from the same flaw. Many companies track daily active users, acceptance rates and session length. Yet heavy usage and negative impact can go together: a tool can reach 100% adoption and still make everyone slower.

PR count is just as much a vanity metric. Faros telemetry from more than 10,000 developers shows teams with high AI adoption merging 98% more pull requests, while review time climbs 91%. Most of those developers used autocomplete rather than agents, and the data is correlational. Merging more PRs is not the same as delivering more value.

What do the measured numbers say?

They are modest, and far from the 10x in board decks.

Stanford’s research on software engineering productivity spans more than 120,000 developers. In a talk on that work, Yegor Denisov-Blanch set 46 teams using AI against 46 similar teams that were not. For that cohort, the median productivity gain came to about 10 percent as of July 2025. Laura Tacho’s DX data, covering 121,000 developers across 450 companies, lands at around 10% as well.

The gap between perceived and measured speed is large. In METR’s randomized controlled trial, 16 experienced open-source developers worked 19% slower with early-2025 AI tools, yet believed they were 20% faster. The sample is small, and METR now flags the result as out of date, but the point is the distance between felt and measured speed. The board only ever sees the felt number. In PwC’s 2026 CEO survey, 56% of CEOs say they have seen no significant financial benefit from AI so far. MIT NANDA’s report says 95% of organizations are getting zero return, a figure Kevin Werbach disputes (opens in a new tab) because the report does not show where it comes from.

Measure output after adoption, and the numbers grow. A study of 300 engineers at one company (opens in a new tab) who used an AI coding assistant recorded a 31.8% drop in PR review cycle time, a 60.1% rise in code shipped and roughly 20 minutes saved per developer each day. In the same Faros data, developers in the high-adoption cohort touch 47% more pull requests a day.

All of these are PR-based metrics, which count output rather than outcome. We think better numbers will arrive over time. Until they do, the best number the customer has beats waiting for the perfect measurement.

What should we measure instead?

Follow the DX AI Measurement Framework and measure along three dimensions, each answering one question:

• Utilization: are people using it? Daily active users, acceptance rates, and how deeply each team has adopted the features. This is what most companies measure. It is necessary, and it is not enough.
• Impact: is delivery getting better? Productivity per team, quality of production code, time to delivery, and how satisfied and confident developers feel. This is harder to measure, and it is what counts.
• Cost: what does it take? Token usage and inference spend, licences per seat, infrastructure, and what you invest in context engineering. Cost decides ROI, and it often gets ignored until the budget review.

On the impact side, the following metrics track what reaches production rather than what passes from person to person:

1. Deployment frequency: code that reaches production, not PRs that get merged.
2. Lead time from idea to first users: DORA measures lead time from commit to production. We start the clock when the idea is logged and stop it when the first users can use the change, so the wait before the first commit counts and a feature behind a closed flag has not arrived.
3. Change failure rate after AI adoption.
4. Rework rate: the share of changes redone after review or acceptance, before they reach users, a measure we defined ourselves. It shows whether exploration and the spec held up. It differs from DORA’s deployment rework rate, which counts unplanned deployments triggered by production incidents.
5. Gate pass rate: how often output gets through the quality gates on the first try.
6. Use: the share of target users active with a feature after 30 days, rather than features shipped.
7. Maintainability and complexity trajectory: could someone else make sense of this code in six months, and is the codebase becoming simpler or harder to change? If every sprint piles on complexity, the velocity is an illusion.
8. Developer satisfaction and confidence.

As a factory matures, automated quality gates take over from manual reviews. The metrics left standing are the ones that track what reaches production and what breaks there.

Why do results differ so much between companies?

DORA’s 2025 report calls AI an amplifier of the strengths and weaknesses an organization already has. On that reading, the same tools and models can make one company much faster and another slower, with the average hiding both. The variance is the signal. Even a well-documented case travels badly. Spotify reports that in the incidents it has reviewed, AI-authored code was not a material direct contributor, but its volume of change grew faster than some of its verification controls could adapt. That is a description of Spotify’s delivery system, and yours is weak in other places.

These are the reasons outcomes diverge:

• “Just turn it on” breeds confusion. Developers use AI differently, code quality swings widely and security gaps show up at random.
• Culture beats tools. Where organizations punish AI-assisted “mistakes”, developers hide their usage entirely.
• No shared context, no shared results. Without structured conventions and documentation, each developer starts from scratch.
• A copied playbook takes no account of your bottlenecks, your codebase or your team dynamics.

In the Stanford talk, a clean codebase predicted the gain better than token spend did: an R-squared of 0.40 for cleanliness, against a correlation coefficient r of about 0.2 for tokens. These are two different statistics. Squared, the 0.2 becomes 0.04, a tenth of the 0.40. Teams burning around 10 million tokens per engineer per month did worse than teams spending a little less, a dip the speaker described as very directional. Our reading: the agent runs out of context it can trust and produces plausible code that collides with conventions the codebase never wrote down.

A clean codebase shows the agent what good looks like, what it must never do, and what gets checked before anything ships. Tests catch what reviewers cannot, the agent can copy naming patterns without being told, architecture boundaries fail loudly, and linting and fitness functions stop bad output before it lands. Buying tokens before you pay down structural debt bends the curve the wrong way. Teams whose gains compound treat codebase health as the operating budget for AI.

The other half is enablement. Tacho says adoption on its own does not guarantee results, and in her words, “This is really a management problem.” By enablement we mean the processes around the work, the review queues, and the room teams have to change how they ship. This is DORA’s amplifier in action: add AI to a slow review process, and faster code piles up in front of a slow review queue. The 300-engineer study saw the same pattern from the opposite end: gains rose with adoption intensity, and low adopters saw little or even negative impact.

Adopting AI is a change initiative, much as Agile was 20 years ago, with the same resistance, the same organizational dynamics and the same need for structured enablement.

Where do the returns come from?

Maintenance work, not the demos. DX Research asked developers to rank ten tasks by how much time they feel AI saves them. Leading by a clear margin: stack trace analysis, refactoring existing code, mid-loop code generation and test case generation, all of them pattern-heavy and rich in context. At the bottom sit code explanation, initial code scaffolding, brainstorming and planning. Code explanation comes last and scaffolding near last, yet those two are what most AI demos show and where most rollouts begin. In our experience, not in the DX data, teams that bring in AI for greenfield development plateau within a few weeks. Teams that bring it in for brownfield work, debugging, refactoring and test generation, are still finding new value six months later.

Code understanding. On our reading, this is a different job from the code explanation at the bottom of DX’s list: having an assistant explain one function is one thing; having a tool read an entire legacy system and write its specification is another. Morgan Stanley says its in-house tool DevGen.AI had saved developers more than 280,000 hours by June 2025. The tool turns code written in old languages into plain-English specifications, which teams can then rewrite. In a legacy system like that, the time goes into reading, understanding and reverse-engineering what already exists, not into writing code.

Automation and risk reduction over time savings. Data from more than 2,500 self-reported use cases (opens in a new tab) shows time savings as the default goal: 35% of use cases target it, and teams cluster between 1 and 10 hours saved per week, most of them around 5. The payoff is modest. Risk reduction accounts for just 3.4% of use cases, yet 25% of those report transformational ROI. Wherever agentic automation comes up, it beats the other categories. Leaders report more transformational impact than individual contributors, with 17% of their use cases in that category, because they choose strategic, high-stakes use cases. Breadth wins too: the more use cases an organization submitted, the better its ROI across the board. Coding-related use cases show higher ROI and less negative impact than the average.

Juniors gain the most. In the same 300-engineer study, a preprint written by the tool’s own builders, junior engineers gained 77% in productivity, mid-level engineers 45% and senior engineers 44%. The leading use cases were UI and frontend development (25.2%), bug fixing and debugging (21.8%) and backend code generation (21.1%).

What ties this together: AI is strong at pattern-heavy, context-rich tasks and mediocre at tasks that demand deep comprehension. The patterns it handles well are the ones it can see in your codebase, and that is why context engineering matters more than prompt engineering.

How long until it pays off?

In our experience, things get worse first: GenAI creates more work before it creates less. The turning point varies from company to company, and your own baseline, taken before the rollout, is what reveals it. Teams that hit the trough without having planned for it give up right there.

What do I answer when the board asks for 10x?

Answer with two questions. Are we measuring at all? And did anything around the work change before we started counting? If the answers are no and nothing, what you measured was adoption without enablement. That, in our reading, is exactly what Tacho’s steady 10 percent shows.

Then begin where your own data lives:

1. Take your baseline first, for the DORA metrics, rework rate, use of what ships and cost per change.
2. Find out where developer time goes before automating anything. Look for what blocks the most people and causes the most rework, and aim AI at that constraint rather than at whatever is easiest to automate.
3. Run a controlled experiment: one team, one codebase, real data.
4. Invest in the codebase and in enablement before tokens: shared context, conventions, tests and gates the agent can pass by itself.
5. Resource for the trough: plan for extra work before the gains arrive, and lay the foundations first.

What this guide cannot tell you

Many of the numbers here come from surveys, self-reported use cases or vendor telemetry, and PR-based metrics count output rather than outcome. The large studies agree on a modest median and a wide spread. Where your team falls on that spread, they cannot say. Only your own baseline can.

Further reading

• Measuring impact →
• Quality and productivity metrics →
• AI productivity research evidence →
• DORA 2025 Report and AI Strategy →
• Where gains and losses occur →
• The learning curve →
• The surrounding quality system →
• Modernization and interface design →
• Agentic Engineering for Engineering and Product Leaders →