Skip to content

Knowledge base

Measuring impact

Measure agentic engineering by pairing AI metrics such as adoption, AI-assisted share and spend with the delivery metrics that predate AI, such as throughput and change failure rate. AI metrics show what is happening. Core metrics show whether it works.

Written by

Updated:

Getting Started 2

Our position. We measure impact, not usage. A change counts once it runs in production and customers use it, pay for it or tell you it helped (opens in a new tab): merged pull requests are output. A gauge without a written red line is decoration (opens in a new tab). We read four measures per team against a baseline from the start: the DORA metrics, with lead time counted from the logged idea to the first users rather than from commit to production; rework rate, the share of changes redone before they reach users; use, the share of target users active with what shipped after 30 days; and cost per change.

Activity metrics are easy to collect and easy to game. A team that counts usage or AI-written code learns how busy it is, not whether delivery improved.

How do you measure AI impact without fooling yourself?

The consensus among companies that measure seriously is to pair AI-specific metrics with the core engineering metrics that predate AI. AI metrics (adoption, share of AI-assisted PRs, spend) show what is happening; core metrics (PR throughput, change failure rate, change confidence, maintainability) show whether it is working, and tracking speed and quality together keeps each metric honest (The Pragmatic Engineer - How tech companies measure the impact of AI on software development (opens in a new tab)). The need is widespread: in LeadDev’s 2025 AI Impact Report, 60% of 880 surveyed leaders cited lack of clear metrics as their biggest AI challenge (The Pragmatic Engineer - How tech companies measure the impact of AI on software development (opens in a new tab)).

A useful taxonomy separates activity metrics (usage, share of AI-generated code), which carry low information value and invite gaming, from outcome metrics (deployment frequency, change failure rate), which measure actual system change; a developer could merge 100 pull requests in a day and still ship one deploy a month, and the team, not the individual, is the unit of delivery (on the DX AI-assisted engineering report (opens in a new tab)).

Tracking “AI code” share is itself an anti-pattern once AI augments rather than replaces the developer (Refactoring - How Engineering Teams use AI (opens in a new tab)). DX groups measurement into utilization, impact, and cost, and recommends combining telemetry with experience data because change confidence cannot be read from system data (DX Q1 2026 AI-Assisted Engineering Impact Report). Once-fashionable metrics like suggestion acceptance rate have largely been abandoned because they measure the moment of suggestion and miss reverts, bugs, and maintainability (The Pragmatic Engineer - How tech companies measure the impact of AI on software development (opens in a new tab)).

Amazon’s engineering leadership drew the line between a measure and a target in a memo to its engineers in October 2026. It calls input metrics such as token usage and AI tool adoption, and throughput metrics such as code changes and reviews, deeply flawed because they are disconnected from outcomes, and warns that as targets they raise cost and complexity: maximizing tokens works against frugality, and counting lines of code works against simplification. It also reports that these inputs correlate well with the outcomes that matter and expects them to keep doing so for some time, so it allows them as a measure used with judgement. The memo publishes no correlation figure, so the correlation is Amazon’s own unquantified claim. Its practical advice is to read the tails of the adoption distribution rather than the average: the 10th percentile, to find the teams not using AI effectively and learn why, and the 90th, to check whether the heaviest users delivered more to match (Amazon - An update from Amazon engineering leadership on the role of the developer in the age of AI (opens in a new tab), October 2026).

How do companies measure AI impact in practice?

Company practice shows what this looks like concretely. Dropbox pairs adoption and CSAT metrics with Core 4 outcomes and finds regular AI users merge 20% more PRs while reducing change failure rate; Microsoft tracks “bad developer days” as a real-time friction signal; Glassdoor measures A/B tests per month as an innovation outcome and tracks a per-tool capacity-utilization rate (“percentage capacity worked”) to spot plateaus; Webflow found the biggest savings among developers with more than three years of tenure, at roughly a 20% PR-throughput lift; Shopify runs an AI leaderboard celebrating its biggest token spenders; Monzo switched off default Copilot code review after concluding it got little value for a lot of LLM usage, while measuring 40-60% effort savings on migrations (The Pragmatic Engineer - How tech companies measure the impact of AI on software development (opens in a new tab)).

Booking.com used segmented cohort data across 3,500 engineers to find that daily users had 16% higher code throughput and that fully adopted teams showed 30% higher throughput, then steered enablement toward the segments getting the least value (Tiatra - How Booking.com measures the impact of AI on developer productivity (opens in a new tab)). Intercom went furthest on automation: an automated code-approval system built from backtesting and human labeling approves 17.6% of changes without a human in the loop, shaped toward safe and simple PRs, which they hold to be compatible with SOC 2, ISO 27001, and HIPAA certification (Intercom talk - Doubling engineering throughput (opens in a new tab)).

Amazon takes the measurement one level higher, translating the whole delivery process into economics. Its Cost to Serve Software (CTS-SW) metric, the total cost of delivering a unit of software, dropped 15.9% year over year in 2024 from developer experience improvements, alongside 18.3% more weekly deployments per builder, 30.4% fewer human interventions per deployment, and 32.5% fewer incident-related tickets per deployment since 2022. The delivery unit adapts to architecture: deployments for microservices, PR completions for monoliths and mobile apps. The framework explicitly rejects velocity-only measures and “minutes saved” arithmetic as gameable and disconnected from business value (Amazon Cost to Serve Software (CTS-SW); Amazon – Business value of developer experience improvements (opens in a new tab), July 2025). The metric-targeting failure hangs over all of this: PRs are output, not outcomes, and any single metric made into a target stops being a good measure (AI Productivity Research Evidence).

How do you measure what AI costs?

Cost is becoming a first-class measurement pillar of its own. The Tata 1mg deployment ran at $30-34 per engineer per month, roughly 1-2% of engineering cost, with LLM API calls making up 91.5% of total spend (Production study - Intuition to Evidence: Measuring AI’s True Impact on Developer Productivity (opens in a new tab)). Pragmatic Engineer’s April 2026 survey of 900+ engineers finds companies budgeting around $100-200 per engineer per month for top-tier plans, about 30% of respondents hitting usage limits, and a regional split in which UK and EU companies scrutinize spend while US companies invest first and measure later; some EU firms now train new joiners on model economics, such as when Claude Sonnet suffices versus Opus (The impact of AI on software engineers in 2026 (opens in a new tab), April 2026).

Developer-token spend and production-agent spend need separate ledgers. Trace developer consumption from each prompt or session through its work item, merge, and deployment. For production agents, divide spend by completed business tasks instead of developer activity. This keeps experimentation cost and production unit economics distinct (WeAreDevelopers – AI Wrote the Code. Now What? (opens in a new tab), July 2026, Harness vendor talk).

Spend does not convert linearly into results: Stanford telemetry shows only a loose correlation between token volume and gains, with a “death valley” around the 10-million-token-per-month mark where heavy usage performs worse than moderate usage; usage quality matters more than usage volume, and their 0-4 AI-practices maturity ladder (from individual use through shared team prompts and rules to agentic orchestration) is detectable from git history (Stanford - measuring AI ROI talk (opens in a new tab)). For the agent era, DX’s whitepaper recommends measuring agents as team extensions, treating every developer increasingly as a lead of a small agent team, and separating production code from disposable AI prototypes; it also cites Intercom nearly doubling adoption with a 41% rise in AI-driven time savings and Block (4,000+ engineers) steering its goose agent with the framework (DX – Measuring AI code assistants and agents (opens in a new tab), vendor methodology).

How do you track change risk when agents write code?

Change risk also needs to move with the work. Recalculate it after each delivery stage from code provenance, repository health, incidents, test evidence, and reviewer independence. Instrument stage-level outer-loop lead time alongside total cycle time, so a faster generation step cannot hide a slower review, verification, or deployment step (WeAreDevelopers – AI Wrote the Code. Now What? (opens in a new tab), July 2026, Harness vendor talk).

Resources

Articles

Talks & Videos