Measure AI impact, not AI usage
Most companies track how much their developers use AI tools. The DX AI Measurement Framework adds impact and cost, and the metrics that survive measure what reaches production.
Written by
-
Benedikt Stemmildt Founder & Co-CEO
Published: 16 April 2026 Updated: 2 min read
Your team adopted Copilot six months ago. Nobody can tell you if it’s working.
They track usage. They should track impact.
The DX AI Measurement Framework
The DX AI Measurement Framework has three dimensions. Miss one, and you’re flying blind.
Dimension 1: utilization
- Daily active users across tools
- Code acceptance and suggestion rates
- Feature adoption depth per team
- Session frequency and duration
This is what most companies measure. It’s necessary but insufficient.
Dimension 2: impact
- Actual productivity gains per team
- Quality changes in production code
- Time-to-delivery improvements
- Developer satisfaction and confidence scores
This is what matters. But it’s harder to measure.
Dimension 3: cost
- Token usage and inference spend
- Licensing fees per seat
- Infrastructure and hosting costs
- Context engineering investment
This determines ROI. It is often overlooked until budget review.
Why utilization alone fails
High usage plus negative impact is a disaster. You could have 100% adoption of a tool that makes everyone slower.
The measurement trap sounds like this:
“Our developers use Copilot 8 hours a day!”
Great. Are they shipping better code faster?
“…we haven’t measured that.”
This is the equivalent of measuring keyboard activity instead of customer value.
What to track instead
- Deployment frequency (code reaching production, not PRs merged)
- Lead time for changes (spec to production, not commit to merge)
- Change failure rate post-AI adoption
- Rework rate (time fixing AI output vs. time AI saved)
- Gate pass rate (how often output clears quality gates on the first attempt)
- Developer satisfaction and confidence scores
PR count is a vanity metric
PR count is a vanity metric. In Faros AI’s telemetry, developers on teams with high AI adoption merge 98% more pull requests. Review time increases 91%. More PRs merged, less value delivered. Goodhart’s Law in action.
As your factory matures, manual reviews get replaced by automated quality gates. The metrics that survive are the ones measuring what reaches production and what breaks there, not what moves between humans.
The teams winning with AI aren’t the ones with the highest adoption numbers. They’re the ones who know what those numbers actually mean for delivery.
(Source: Faros AI, The AI Productivity Paradox Report 2025 (opens in a new tab), July 2025, telemetry from more than 10,000 developers on 1,255 teams.)
Further reading
Written by
-
Benedikt Stemmildt · Founder & Co-CEO
More on Measuring AI impact
- Clean codebases decide who gets value from AI Stanford research on 120,000 developers, presented in a talk, found codebase cleanliness predicted AI value better than token spend. Teams with weak codebases regress past a certain spend.
- AI inflates the metrics that don't measure value AI pushes up every number that is easy to measure: code volume, pull requests, velocity. Four metrics show whether the value went up with them.
- Morgan Stanley saved 280,000 hours by reading old code Most teams point AI at code generation. Morgan Stanley pointed it at code understanding, the step where developer time actually went.
Test the constraint in your system
The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.
Training on this topic: Agentic Engineering for Engineering and Product LeadersWant to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check