Skip to content

Blog

Measure AI impact, not AI usage

Most companies track how much their developers use AI tools. The DX AI Measurement Framework adds impact and cost, and the metrics that survive measure what reaches production.

Written by

Measuring AI impact

Published: 16 April 2026 Updated: 2 min read

Your team adopted Copilot six months ago. Nobody can tell you if it’s working.

They track usage. They should track impact.

The DX AI Measurement Framework

The DX AI Measurement Framework has three dimensions. Miss one, and you’re flying blind.

Dimension 1: utilization

  • Daily active users across tools
  • Code acceptance and suggestion rates
  • Feature adoption depth per team
  • Session frequency and duration

This is what most companies measure. It’s necessary but insufficient.

Dimension 2: impact

  • Actual productivity gains per team
  • Quality changes in production code
  • Time-to-delivery improvements
  • Developer satisfaction and confidence scores

This is what matters. But it’s harder to measure.

Dimension 3: cost

  • Token usage and inference spend
  • Licensing fees per seat
  • Infrastructure and hosting costs
  • Context engineering investment

This determines ROI. It is often overlooked until budget review.

Why utilization alone fails

High usage plus negative impact is a disaster. You could have 100% adoption of a tool that makes everyone slower.

The measurement trap sounds like this:

“Our developers use Copilot 8 hours a day!”

Great. Are they shipping better code faster?

“…we haven’t measured that.”

This is the equivalent of measuring keyboard activity instead of customer value.

What to track instead

  1. Deployment frequency (code reaching production, not PRs merged)
  2. Lead time for changes (spec to production, not commit to merge)
  3. Change failure rate post-AI adoption
  4. Rework rate (time fixing AI output vs. time AI saved)
  5. Gate pass rate (how often output clears quality gates on the first attempt)
  6. Developer satisfaction and confidence scores

PR count is a vanity metric

PR count is a vanity metric. In Faros AI’s telemetry, developers on teams with high AI adoption merge 98% more pull requests. Review time increases 91%. More PRs merged, less value delivered. Goodhart’s Law in action.

As your factory matures, manual reviews get replaced by automated quality gates. The metrics that survive are the ones measuring what reaches production and what breaks there, not what moves between humans.

The teams winning with AI aren’t the ones with the highest adoption numbers. They’re the ones who know what those numbers actually mean for delivery.

(Source: Faros AI, The AI Productivity Paradox Report 2025 (opens in a new tab), July 2025, telemetry from more than 10,000 developers on 1,255 teams.)

Further reading

  1. Clean codebases decide who gets value from AI Stanford research on 120,000 developers, presented in a talk, found codebase cleanliness predicted AI value better than token spend. Teams with weak codebases regress past a certain spend. 2 min readShort post
  2. AI inflates the metrics that don't measure value AI pushes up every number that is easy to measure: code volume, pull requests, velocity. Four metrics show whether the value went up with them. 1 min readShort post
  3. Morgan Stanley saved 280,000 hours by reading old code Most teams point AI at code generation. Morgan Stanley pointed it at code understanding, the step where developer time actually went. 2 min readShort post

All 16 posts on Measuring AI impact

ClientsClient work

Test the constraint in your system

The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.

Training on this topic: Agentic Engineering for Engineering and Product Leaders
See the assessment

Want to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check