AEMM industry evidence

Our position. We hold that usage is not ROI (opens in a new tab): a team can use the tool heavily and ship no faster, because the constraint was never the typing. We measure our own baseline (opens in a new tab) instead of copying another company’s playbook, because AI outcomes vary too much between organizations to transfer.

A Stanford research group with more than 100,000 engineers across more than 600 companies in its data set puts the average net productivity gain from AI at roughly 15 to 20%, once the rework on bugs the AI introduced is subtracted (Does AI Actually Boost Developer Productivity? (opens in a new tab)). In a later talk the same group matched 46 teams using AI with 46 similar teams that were not and found a median net gain of about 10%, with the gap between top and bottom teams widening (Can You Prove AI ROI? (opens in a new tab)).

What does industry evidence say about each AEMM level?

External Maturity Models Mapping to AEMM Levels

• A five-level practitioner model maps closely: L1 (10% automation, Copilot) > L2 (60-70%, multi-threading agents) > L3 (80-90%, 2-3 person teams) > L4 (95-98%, platform teams + 47 agents) > L5 (99%, strategic oversight only): “ProgrammierEnde (opens in a new tab)” – Mayflower
• OpenAI’s three-tier model: Delegate (routine work agents handle), Review (human assessment required), Own (decisions requiring judgment) – agents become first-pass implementers, engineers become reviewers: “Building an AI-Native Engineering Team (opens in a new tab)” – OpenAI Codex
• GitHub’s 8 pillars framework for AI adoption: AI Advocates, Clear Policies & Guardrails, Learning & Development, Data-Driven Metrics, Dedicated Leadership (DRI), Executive Support, Right-Fit Tooling, Communities of Practice: “AI-Powered Workforce Playbook (opens in a new tab)” – GitHub
• Agent sophistication levels from static > reactive > deliberative > learning > self-evolving autonomous, with MCP replacing traditional APIs as integration standard: “Die neue Architektur (opens in a new tab)”
• One capacity-doubling analogy for AI agents estimates a doubling every 7 months for general tasks and every 70 days for code tasks, or 4-6 doublings per year in code-task capacity: “Devin 2.0 and the Future of SWE (opens in a new tab)” – Cognition
• The “agentic layer” grades the meta-code wrapped around a codebase that teaches agents to operate it, and grades it 1-5: Grade 1 (CLAUDE.md plus a prime command), Grade 2 (specialized prompts, specs directories, sub-agents), Grade 3 (agents.md coordination files), Grade 4 (closed-loop prompts with request-validate-resolve cycles), Grade 5 (multiple specialized prompts, MCP servers, skills). The thesis, the “codebase singularity”, is that once this layer is powerful enough, agents operate the codebase more effectively than any individual engineer. It grades the harness rather than the engineer, so it complements AEMM rather than restating it: “The Codebase Singularity (opens in a new tab)”

Concrete Metrics Per Level

• At basic AI adoption (AEMM L2): even with optimistic assumptions (40% cycle time coding, 60% AI-supportable, 55% faster), overall time savings only ~13% of total delivery cycle: “XP Keynote - AI for teams (opens in a new tab)”
• At inner loop agent level (AEMM L3-L4): 40-50% productivity gain in implementation, though debugging time may increase: “Agents in the Outer Loop (opens in a new tab)” – OpenHands
• At full modernization (AEMM L4-L5): 10-20x speed, 90-95% automation, 3 months vs. 18 months traditional, ~200k EUR + 30k EUR AI vs. 900k EUR traditional: “Modernisierung mit AI (opens in a new tab)”
• An engineering team reports rebuilding five years of legacy software in eleven months with one third of the original headcount, preceded by a two-week AI analysis pass across fifteen legacy projects, and its lead engineer reports writing no code by hand since January; the episode headlines the delivery as 72 times faster than the traditional path. Single-company self-report on a rebuild with a known target, published by a consultancy that sells the transition, and no independent audit: “Integrating AI into development processes (opens in a new tab)”, 18 May 2026
• High adoption cohort (>75th percentile): 61.3% code volume increase vs. low adoption cohort (<25th percentile): 11.4% decline – 72.7 percentage point differential demonstrates engagement intensity matters: “Measuring AI’s True Impact (opens in a new tab)”
• Booking.com: daily AI users achieved 16% higher code throughput; developer satisfaction increased 15 points over six months; fully adopted teams showed 30% higher throughput: “Booking.com Measures AI Impact (opens in a new tab)” – Tiatra
• Dropbox: 90% AI adoption but real value emerged correlating this with 20% increase in merged PRs while reducing change failure rates: “How Tech Companies Measure AI Impact (opens in a new tab)” – The Pragmatic Engineer

Team Sizing Data Across Levels

• Team sizes shrink while output multiplies: 7 people (L2) > 3 people (L3) > ops teams (L4+): the five-level practitioner model
• Human roles evolve across levels: coders (L1-L2) > AI orchestrators (L3) > strategists (L4-L5): the five-level practitioner model
• Traditional agile practices become obsolete at higher maturity levels; infrastructure scales dramatically with 100x AI workloads at Level 4: the five-level practitioner model
• “A single developer can do the work of five developers a few years ago when properly leveraging AI systems”: “Compound Engineering (opens in a new tab)” – Every
• Every runs five products, each primarily built and run by one person and used by thousands of people every day: Compound Engineering

Measurement Approaches from Industry

• 18 companies studied layer AI-specific metrics on top of proven developer productivity measurements (Change Failure Rate, PR throughput) rather than starting fresh: the Pragmatic Engineer survey
• Three-dimensional measurement framework: utilization (adoption rates), impact (productivity gains and quality metrics), cost (ROI and governance): “Measuring AI Code Assistants and Agents (opens in a new tab)” – DX
• DX Q1 2026 AI Measurement Framework adds a concrete “Core 4” metric set: PR throughput, perceived rate of delivery, Developer Experience Index (DXI), code maintainability, change confidence, change fail %, plus Human-Equivalent Hours (HEH) for autonomous agents. It converges, independently and across 400+ companies, on the same four KPIs in Measuring Agentic Engineering Impact: see DX Q1 2026 Impact Report – DX
• Industry benchmark (Q1 2026, 400+ companies): 93% AI adoption, 3.9 hrs/week avg time savings (4.72 daily users), 27.4% AI-authored code, median PR throughput 2.4/week for daily users vs. 1.5 for non-users. Change Failure Rate swung by ~2pp at some firms — ~50% more defects. Outcome depends on enablement and measurement, not tool choice: DX Q1 2026 Impact Report
• Measurement progression: Phase 1 (breadth: MAU/MEU) > Phase 2 (depth: user segmentation) > Phase 3 (business impact: cycle time, quality, wellbeing): GitHub AI-Powered Workforce Playbook
• Microsoft measures “bad developer days”; Glassdoor tracks experimentation frequency; Monzo carefully avoids AI in customer-data areas despite efficiency potential: “How Tech Companies Measure AI Impact (opens in a new tab)”
• Companies analyze by comparing AI users vs. non-users, before/after adoption, and cohort analysis; Webflow discovered engineers with 3+ years tenure saved most time: “How Tech Companies Measure AI Impact (opens in a new tab)”
• Five automated git-based quality proxies: large commits (<20% target), sprawling commits (<10% target), test-first discipline (>50% target), files per commit (<5 target), lines per commit (<100 target): “PDCA Framework for AI Code Generation (opens in a new tab)” – InfoQ
• Google DORA 2024 reports that every 25% AI adoption increase correlates with a 7.2% delivery-stability decrease, and GitClear reports a tenfold increase in duplicated code blocks. A separate clone study cited in the same article puts the defect rate of cloned code at 17%: PDCA Framework for AI Code Generation

Stanford 120k Developers Study (ROI Evidence)

• A later talk by the research group behind the 100k developers study below: 46 teams using AI matched with 46 similar teams without AI, net productivity gain measured quarterly. The median stood at about 10% as of July 2025. The gap between top performers and the rest is widening, a “rich get richer” effect in which early adopters might compound their gains while strugglers fall further behind: “Can You Prove AI ROI? (opens in a new tab)” – Stanford
• An experimental environment cleanliness index correlates with AI productivity gains (R-sq 0.40), which the speaker calls “a pretty decent correlation”; the index combines tests, types, documentation, modularity and code quality, and the talk’s takeaway is to invest in codebase hygiene. AI token spend correlates loosely (R-sq about 0.20) – quality of use matters more than volume: the Stanford study
• Activity metrics (lines of code, PR counts, DORA) that say “more” without proving “better” fail to demonstrate value. Recommended: engineering output as the primary metric, with rework and refactoring, quality and risk, and people and DevOps metrics as guardrails: the Stanford study. Self-assessment is unreliable: in the group’s earlier experiment with 43 developers, people misjudged their productivity by about 30 percentile points: Stanford 100k Developers Study (opens in a new tab)
• Progression from no AI use (Level 0) to agentic orchestration (Level 4). The talk projects the widening gap between top and bottom teams forward, by its own account “very unscientifically and very illustratively”, without a number: the Stanford study

McKinsey Enterprise Transformation Data

• Most enterprises plateau at 5-15% AI productivity gains. Top performers achieve 7x better results with AI-native workflows and 6x higher role restructuring rates. 70% of surveyed companies have not changed their role definitions: “Moving away from Agile (opens in a new tab)” – McKinsey
• Pod structure shift: 3-5 person pods replace 8-12 person teams. Spec-driven development replaces story-driven: PMs iterate specs with agents before developers write code. Continuous planning replaces quarterly sprints: McKinsey
• Bank case study: 51% increase in code merges and 60x higher agent consumption after restructuring. This shift is structural – comparable to waterfall-to-agile (3-6 months redesign): McKinsey
• Bottom performers lack basic output metrics: “only 10% were measuring productivity”: McKinsey

Agent Autonomy Patterns

• Three pillars of autonomy: frontier models, advanced context management, exhaustive verification. Must invest in all three. Replit Agent evolved from single ReAct-style to multi-agent system (Manager, Editor, Verifier). “Ralph mode” achieves 90% autonomy rates: “Autonomy Is All You Need (opens in a new tab)” – Replit
• Decision-time guidance: inject situational instructions at key moments rather than front-loading all rules. “We don’t strive for full autonomy. We want the user to stay involved and engaged”: the Replit talk
• Compounding errors are the key challenge: “The longer the trajectory, the more likely the agent makes an error.” Multi-agent architecture where each agent performs the smallest possible task to reduce error probability: the Replit talk
• Compounding Engineering: Plan > Delegate > Assess > Codify. 15 people managing 4 products. “A single developer can do the work of five developers a few years ago”: “Dispatch from the AI-native Future (opens in a new tab)” – Every

Practitioner Evidence

12-Factor Agents: Mapping to AEMM Levels

• Agents are software, not magic. Most production agents are “not that agentic at all” – they are mostly deterministic software with small LLM loops of 3-10 steps (“micro agents”). The pattern that works: a mostly deterministic DAG with very small agent loops embedded at decision points. This maps to AEMM L2-L3 – teams building reliable agents before pursuing full autonomy: “12-Factor Agents (opens in a new tab)” – AI Engineer, 3 July 2025
• Own your prompts, context window, and control flow. The three pillars of agent reliability: (a) write every token of your prompts by hand once you need quality, (b) control what goes on the context window (don’t blindly append errors/stack traces), (c) own the loop that determines when/how/why you exit. This is the L2-to-L3 transition criteria in practice: 12-Factor Agents
• “LLMs are pure functions, tokens in, tokens out, and everything in making agents good is context engineering.” Context engineering is the single lever teams control (short of retraining). This validates context engineering as the primary skill differentiating AEMM levels: 12-Factor Agents
• “Not every problem needs an agent.” The discipline to recognize when a bash script suffices in 90 seconds vs. spending 2 hours prompt-engineering an agent is a core competency. Knowing when not to use an agent marks the higher maturity levels, L3 and above: 12-Factor Agents

Claude Code SDK Custom Agents: Level 4-5 Patterns

• Progression path: Better Agents (L2) -> More Agents (L3) -> Custom Agents (L4-5). “Out-of-the-box agents are built for everyone’s codebase, not yours.” Custom agents optimize for your domain, your problems, your edge cases: “Agentic Coding ENDGAME: Claude Code SDK Custom Agents (opens in a new tab)”
• The system prompt is “the most important element of your custom agents with zero exceptions.” Overwriting Claude Code’s system prompt completely replaces the product – it is no longer Claude Code, it is a custom agent. This is L4 behavior: building domain-specific agents, not configuring generic ones
• Core Four to track at every level: Context, Model, Prompt (system + user), Tools. Running /tools reveals all available tools (15+ built-in) plus MCP tools. These all consume context window space even when unused. Custom agents at L4-5 strip unnecessary tools to reduce noise and improve focus
• Micro-SDLC Agent: Multi-agent workflow with Planner -> Builder -> Reviewer stages, orchestrated through a web UI with WebSocket real-time updates. Each agent hands off to the next stage. The task board pattern (plan -> build -> review -> ship) is a concrete L4 implementation. “Agent coding is not so much about what we can do anymore. It’s about what we can teach our agents to do”

Agent Experts: Self-Improving Systems (Level 4-5)

• Self-improving agents with expertise files: An “agent expert” is a self-improving template metaprompt with a YAML expertise file that functions as the agent’s mental model. Unlike memory files (static, manually updated), expertise files are updated automatically by the agent after every task via a self-improve step: “Agent Experts (opens in a new tab)” – Agentic Horizon lesson
• “The difference between a generic agent and an agent expert is simple. One executes and forgets, the other executes and learns.” This is the L4/L5 distinction: L4 agents operate autonomously, L5 agents compound their own capability over time
• Plan-Build-Self-Improve workflow: Three-step cycle where step 3 (self-improve) automatically updates the expertise file with what changed. The agent keeps track of changes, validates its mental model against the codebase, and accumulates expertise without human intervention. L5 behavior: the agent gets better at its job without human coaching
• Meta-agentics: prompts writing prompts, agents building agents, skills building skills. “There is no codebase I create that does not have meta-agentics.” This is the L5 Code Multiplier operating principle – the system generates its own components
• Scale agents against problems, not roles: Instead of one agent answering a question, deploy three or five with the same prompt. “Sometimes if you throw five agents at the problem, only one makes it.” Compose results for higher confidence

Continual System Prompt Learning (Level 3 Enabler)

• 5-15% improvement from 150 examples: On SWEBench, Claude Code improved by 5% more GitHub issues resolved, Cline improved by 15%. No fine-tuning, no model changes – purely system prompt iteration via LLM-as-judge evals with explanations: “System Prompt Learning with Evals (opens in a new tab)”
• Prompt learning vs. RL: Instead of scalar rewards, prompt learning provides English feedback explaining why something failed and what concepts were missed. This is accessible to any engineering team – no data science team required. The L3 transition: teams running structured eval loops to improve agent configurations rather than ad-hoc tweaking
• “Writing really good evals is how you get the best insight into what you could do to improve your agents.” Eval quality is the critical factor. The explanation from the eval drives improvement, not the score

Specific Metrics: Real-World AEMM Level Indicators

• Stanford R-squared 0.40 for environment: An environment cleanliness index of tests, types, documentation, modularity and code quality shows “a pretty decent correlation” with AI productivity gains, so the talk advises investing in codebase hygiene. AI token spend shows a loose R-squared of about 0.20 – quality of use matters more than volume. This metric quantifies the L2-to-L3 transition: “Stanford 120k Developers ROI Study (opens in a new tab)”
• Atlassian 45% PR cycle time reduction with AI-assisted first-pass review. Developer satisfaction went from 49% to 78% after implementing developer joy measurement. These are concrete L3 metrics: “State of Developer Experience 2025 (opens in a new tab)” – Atlassian
• Booking.com: daily AI users achieved 16% higher code throughput; developer satisfaction increased 15 points over six months; fully adopted teams showed 30% higher throughput. This is L2 to L3 transition data
• METR 19% slower: Counterintuitive finding explained by the overhead of context-switching between human review and agent execution – the loop is too tight for complex tasks but too loose for simple ones. This supports the “Elastic Loop” concept where loop tightness should be dynamically adjustable: “The Elastic Loop (opens in a new tab)”
• Stanford 100k developers: more than 100,000 engineers across more than 600 companies. AI coding raises the code delivered by roughly 30 to 40%, but fixing the bugs it introduces brings the average net productivity gain down to roughly 15 to 20%. The group’s earlier ghost engineer finding: roughly 10% of the engineers in its data set at the time did basically no work: “Stanford 100k Developers Study (opens in a new tab)”

AEMM Level Mapping from Claude Code Ecosystem Videos

• Level 1 (Ad-Hoc Adopter): Basic prompting, no context management, single-shot interactions, copy-paste workflow
• Level 2 (Systematic Collaborator): CLAUDE.md, slash commands, explore-plan-execute workflow, XML structured prompts, context window awareness
• Level 3 (Strategic Orchestrator): Context priming, hooks for observability, sub-agent delegation, context bundles, LSP integration, system prompt learning via evals, the RPI workflow (Research-Plan-Implement)
• Level 4 (Autonomous Partner): Multi-agent delegation, background agents, custom SDK agents, skills as composition units, out-of-loop coding, agent experts with self-improving expertise files
• Level 5 (Code Multiplier): Template meta-prompts, agent pipelines, custom domain agents, automated eval systems, meta-agentics (agents building agents), “compute orchestrating compute”

12-Factor Agents: Production Patterns Expanded (Full Talk)

The full 12-Factor Agents talk adds concrete production patterns to the pillars above, each mapped below to the AEMM level it fits.

• Tool use is JSON + code, not magic (L2 insight): “I’m going to go ahead and go on a limb here and say tool use is harmful. And I put it in quotes because I’m not talking about giving an agent access to the world… what is happening is our LLM is putting out JSON. We’re going to give that to some deterministic code that’s going to do something… There’s nothing special about tools. It’s just JSON and code.” Demystifying tool use is the L1-to-L2 transition: “12-Factor Agents (opens in a new tab)” – AI Engineer, 3 July 2025
• Micro agents are the production pattern (L2-L3): “Most production agents weren’t that agentic at all. They were mostly just software.” The pattern that works in production: “a mostly deterministic DAG and you have these very small agent loops with like three to 10 steps.” The cited deploy bot is deterministic CI/CD until it needs a decision, then a small agent loop handles it, then returns to deterministic code. “100 tools, 20 steps, easy. Manageable context, clear responsibilities”: 12-Factor Agents
• Pause/resume via state serialization (L3-L4): “We’re going to allow our agent to call long-running tools. So we can interrupt the workflow, serialize that context window straight into a database because we own the context window… and then when we launch the workflow, eventually it’s going to call back with that state ID and the result. We use the state ID to load the state back out of the database… The agent doesn’t even know that things happened in the background.” This is the L3-to-L4 capability: agents that persist across sessions and resume without context loss: 12-Factor Agents
• Find the bleeding edge (L3+ assessment): “Find something that is right at the boundary of what the model can do reliably – like that it can’t get right all the time – and if you can figure out how to get it right reliably anyways because you’ve engineered reliability into your system, then you will have created something magical and you will have created something that’s better than what everybody else is building.” This is the L3 differentiator: not using agents where they trivially succeed, but engineering reliability at the frontier: 12-Factor Agents (quoting NotebookLM team)
• Contact humans with tools, not output parsing (L3): “Almost everybody is avoiding this very important choice at the very beginning of output where you’re deciding between tool call and message to the human… if you can push that emphasis to a natural language token, you can give the model different ways. You can be ‘I’m done’ or ‘I need clarification’ or ‘I need to talk to a manager.’” Human-in-the-loop is a first-class agent capability, not an escape hatch: 12-Factor Agents

The Elastic Loop: Agency as an AEMM Enabler

The INNOQ talk introduces agency (the ability to make use of autonomy) as a critical human factor in AEMM level progression.

• Loop tightness as maturity indicator: The spectrum runs from tight loops (autocomplete, chain of small steps – high cognitive load, maximum control) to loose loops (async delegation, ambient agents listening for domain events – lower cognitive load, context must be pre-positioned). “When stakes are high and the ambiguity are high… choose a tight loop. And when you’re able to provide the context on demand… the agent just listens in the background.” L1-L2 engineers default to tight loops; L3+ engineers dynamically select loop tightness based on risk and context quality: “The Elastic Loop: Context-Switching (opens in a new tab)” – INNOQ
• Agency vs. autonomy: “Not all people can deal with an overly big amount of autonomy. That’s where agency comes in. Agency is the ability to make use of autonomy. If your people are not agentic… they won’t use your autonomy. You can bury them in autonomy. They won’t embrace it.” High-agency engineers see outsized gains; the “great divide” is along agency, not just skill level. Agency is a human factor that a skill-based maturity level does not capture: The Elastic Loop
• Head work over hand work: “Don’t base your career on your hands work. Base it on your head’s work because the work you’re doing with your hands will get way cheaper.” The baseline skill to develop: “provide judgment, high-level strategy, and handle the weird cases.” This aligns with the Knowledge Shift thesis in Engineering Philosophy and maps to the progression from L1 (coder) to L4-L5 (strategist): The Elastic Loop
• Four strategic questions for loop selection: (1) How volatile is the thing you’re working on? (2) What’s the freshness of the context? (3) Where does trust live in your organization? (4) Which metrics detect drift in your loops? These map directly to AEMM assessment criteria – teams at L1-L2 cannot answer these questions; L3+ teams use them to calibrate daily work: The Elastic Loop
• The decade timeline validates slow level progression: “The problems are tractable, they’re surmountable, but they’re still difficult. If I just average it out, it just feels like a decade to me.” Current models are “savant kids. They have perfect memory of all this stuff. They can convincingly create all kinds of slop that looks really good. But I still think they don’t really know what they’re doing.” This reinforces the AEMM principle that most enterprise teams deliver massive value at Level 3 and should not rush to Level 5: “Why It’s the Decade of Agents (opens in a new tab)”

Which external maturity models map to AEMM?

Two external models cover the same ground from a different angle. Use them to cross-check a level assessment, not to replace it.

The Dark Factory model (see Dark Factory Patterns):

Read the numbers carefully. The Dark Factory model frames its results as a “100x productivity” gain. Generic multipliers do not survive contact with a real delivery pipeline. Take the patterns; express gains through the four KPIs in Measuring Agentic Engineering Impact.

Anthropic’s modernization maturity model has four levels (Anthropic - Code Modernization Playbook (opens in a new tab)):

The mapping lets a modernization program and an AEMM assessment share one maturity conversation without treating the models as identical.

A ten-dimension rubric scores the ladders separately rather than resolving an organisation to one level. Its dimensions are mode of thinking, prompting, human-AI trust and QA, heartbeat agent invocation, model routing, spend absorption, sovereignty and model control, parallelization, AI impact measurement, and governance of AI rollouts, each with eight levels, self-assessed, free to use and drawn from the author’s advisory work since 2022 (AI Maturity Rubric (opens in a new tab)). Three of those dimensions have no AEMM equivalent: spend absorption, sovereignty over which models the organisation controls, and the governance of a rollout. Use it where an assessment has to speak to procurement and legal as well as to engineering, and keep AEMM for the engineering conversation.

Related

• Measuring impact: how to set up the baseline this evidence is compared against.
• AI productivity studies: METR, Stanford, DORA and Multitudes: the studies behind the headline numbers.
• The DX AI-assisted engineering impact report: the benchmark figures cited above, in full.
• Moving away from Agile: McKinsey’s AI operating-model research: why restructured teams outperform tool rollouts.
• Quality and productivity metrics: the measures that separate output from outcome.

Resources

Maturity models and adoption evidence

• AI Maturity Rubric (opens in a new tab) - Free self-assessment scoring ten dimensions of organizational AI adoption across eight levels each, including spend, sovereignty and governance.