AI inflates the metrics that don't measure value
AI pushes up every number that is easy to measure: code volume, pull requests, velocity. Four metrics show whether the value went up with them.
Written by
-
Benedikt Stemmildt Founder & Co-CEO
Published: 8 April 2026 Updated: 1 min read
Your team shipped 3x more code last quarter. Your customers noticed nothing.
Lines of code: up 300%. Pull requests: up 200%. Deploy frequency: up 150%.
You’ve never shipped less value.
The vanity metrics trap
AI inflates everything that’s easy to measure:
- Code volume goes up because generating code is cheap.
- PR count goes up because smaller, faster changes feel productive.
- Task completion goes up because easy tasks get picked first.
- Story point velocity goes up because points measure effort, not impact.
Leadership sees dashboards going up and celebrates. The dashboards are lying.
What goes unmeasured
- Features customers actually use
- Code that survives 6 months without rewrite
- Time spent fixing AI output vs. time AI “saved”
- Technical debt created per feature shipped
- Rework rate on AI-generated code
Where the other 1.9x went
Team ships 3x more code. Users see 1.1x more useful features.
Where did the missing 1.9x go? Into code that needed immediate fixing. Into features no one asked for. Into complexity that slows everything else down.
Four metrics to track instead
- Value throughput. Features multiplied by usage multiplied by retention. Not features shipped.
- Rework rate. Time fixing AI output divided by time AI “saved”. Above 1, fixing the output costs more time than the AI saved.
- Maintainability. Can someone else understand this code in 6 months? AI-generated code often fails this test.
- Complexity trajectory. Is the codebase getting simpler or harder to change? If every sprint adds complexity, velocity is an illusion.
Productivity is not value
If AI delivers massive productivity gains, why aren’t outcomes massively better?
Because productivity is not value. AI is great at productivity. It’s neutral on value.
The gap is context engineering: the practice of ensuring AI builds the right things, not just more things.
Are you measuring velocity, or measuring direction?
Further reading
Written by
-
Benedikt Stemmildt · Founder & Co-CEO
More on Measuring AI impact
- Measure AI impact, not AI usage Most companies track how much their developers use AI tools. The DX AI Measurement Framework adds impact and cost, and the metrics that survive measure what reaches production.
- Morgan Stanley saved 280,000 hours by reading old code Most teams point AI at code generation. Morgan Stanley pointed it at code understanding, the step where developer time actually went.
- Another company's AI success story won't transfer to yours DORA research shows AI returns vary widely between organizations using the same tools. The variance is the signal: measure your own baseline and your own bottlenecks before copying a playbook.
Test the constraint in your system
The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.
Training on this topic: Agentic Engineering for Engineering and Product LeadersWant to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check