Skip to content

Knowledge base

Context and productivity

AI coding tools are limited less by their harness than by the context fed into them, and measured productivity gains sit far below vendor claims. Controlled studies and longitudinal benchmarks show modest throughput gains, and AI amplifies a team's existing engineering quality rather than replacing it.

Written by

Updated:

Tools 4

Our position. We hold that the real constraint is context management, not model intelligence (opens in a new tab). AI multiplies what a team already has, so fix the foundation first, then multiply it (opens in a new tab).

Why is context the bottleneck for AI coding tools?

If the harness has stopped differentiating, the material that goes into it has not. The Baz review-agent builder puts a ceiling on prompting alone (roughly 30-60% of the result) and argues that collecting the codebase context is the hard, defensible part: “prompting everybody can do”, but the context is “the secret sauce of your company” and hard to replicate (Baz on building a code-review agent (opens in a new tab), June 2025). The Context Development Lifecycle talk generalizes this into a four-stage lifecycle: every agent session is a new hire that starts from zero, so the implicit knowledge a senior carries as muscle memory has to be made explicit, and context deserves the same engineering rigor as code.

The Context Development Lifecycle has four stages, generate, evaluate, distribute, and observe, and presses two points hardest: context needs tests, because “every evaluation failure is a specification you didn’t write”, and it needs distribution as a versioned, secured package the way npm and pip made code reusable. Bigger context windows do not dissolve the problem; loading everything just multiplies contradictions, shifting the work from curation to governance (The Context Development Lifecycle (opens in a new tab), February 2026).

The practical corollary is that what is good for human contributors is good for agents. Engineering foundations carry more weight once agents write the code. A DX conference talk argues teams should standardize on industry-standard tooling precisely because it is what sits in the training set; custom package managers and obscure languages make the agent fight its own training. The highest-leverage investment is deterministic validation with clear error messages, since an agent “cannot divine what you mean by a 500 internal error with no other message”, and that validation has to run at dev-time in about 30 seconds rather than only in a 15-20 minute CI job, because agents loop run-test-run-test several times and slow feedback shreds the gain (DX – No-regrets investments for the agentic era (opens in a new tab), December 2025). The same talk names a failure mode teams keep hitting: agents write fake tests against untestable code (“I pushed the button and the button pushed successfully; test passed”), so refactoring legacy code for testability and reason-ability comes before pointing agents at it.

Practitioner demos show context files behaving less like static config and more like living, layered artifacts. In a Google talk fixing a live issue in Gemini CLI’s own repository, a practitioner combined three markdown layers on each run: a personal workflow file (plan mode, branch naming, test-driven steps), a team-maintained best-practices file that pins conventions down to preferred commands (npx vitest over npm test), and generated project-structure files that act as a “city guide” to the codebase. The files were themselves largely LLM-written: after a session the agent reflects on where it got stuck and appends the lessons back to the markdown, so the next run starts smarter, an instance of the talk’s generate-and-observe loop in practice.

The talk’s framing is that with simple tool calling underneath, most of the steering happens in the natural-language instructions rather than the harness, and that markdown beats a validatable config format precisely because non-developers can write it, models are good at synthesizing it, and its fuzziness lets one instruction generalize across tasks; Gemini CLI can also read a project’s existing CLAUDE.md or Cursor rule files rather than forcing its own (Google on Gemini CLI (opens in a new tab), June 2025).

Most of that context describes the code as written. A newer category adds how the code behaves in production. Hud runs a sensor beside the application that records errors and performance per service and per function, captures execution paths, parameters and machine state when an error or slowdown fires, and shows those numbers next to the function in VS Code, JetBrains or Cursor, so an engineer or agent editing a function sees its call volume, latency and error rate first. It supports Node.js, TypeScript, Python and Java. Its customer stories claim incident investigations shrinking from days to minutes, vendor claims without a published method (Hud (opens in a new tab), as of September 2026).

What does the evidence say about productivity?

The measured picture is more modest than vendor claims. The METR randomized study found 16 experienced open-source developers were about 19% slower with AI tools (primarily Cursor) on their own repositories while predicting they would be about 20% faster; the rarely cited methodological caveat is that participants started with a 30-minute introduction and onboarding call that included basic Cursor training, and METR cannot rule out learning effects beyond 50 hours of Cursor use (METR paper, appendix G.3 (opens in a new tab); METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab)). DX’s Q1 2026 benchmark across 400+ companies shows 92.7% adoption, 3.9 hours of self-reported weekly time savings, and 27.4% AI-authored code, but its longitudinal study found PR throughput gains of only 10-15% over more than a year, substantially lower than self-reported savings imply (DX Q1 2026 AI Impact Report).

Tool choice does show up in the benchmark data: daily Cursor users shipped 4.1 PRs per week (up 46% quarter over quarter), Claude users above 4.0, GitHub Copilot users 3.61, and Tabnine users 1.83, with newer AI-native and agentic tools leading by a widening margin (DX Q1 2026 AI Impact Report). Field experience adds texture the benchmarks miss. Rotational Labs’ six-person team tested Cursor and Copilot for three months and found overdelegation backfires, well-scoped narrow tasks work, and accelerating an individual is not the same as accelerating a team; the tools made individual work faster while making collaboration harder, forcing the team to add deliberate synchronization time (Rotational Labs - What We’ve Learned From 3 Months Living with AI Coding Tools (opens in a new tab)).

The convergent conclusion across studies supports building the foundation first: AI amplifies existing engineering quality. Mature teams accelerate, immature teams decelerate, and structured enablement and measurement, not tool rollout, determine outcomes (AI Productivity Research Evidence, DX Q1 2026 AI Impact Report). The 2025 DORA report puts the amplifier framing bluntly: agentic AI “magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones” (Your AI Coding Assistants Will Overwhelm Your Delivery Pipeline (opens in a new tab), January 2026).

That amplification exposes a bottleneck downstream of code generation: the delivery pipeline. With about 90% of developers now using AI to write code faster and DX reporting that daily AI users ship 60% more pull requests, output collides with pipelines built for lower volumes, so work piles up at manual merge, test, and deployment gates and software ships later, not sooner. DORA 2025 finds 77% of organizations deploy once per day or less and 20% only monthly or quarterly, while the top-performing 23% deploy dozens to hundreds of times per day; the recommendation is to strengthen test-driven development, trunk-based continuous integration, and continuous deployment before scaling AI, not after (Your AI Coding Assistants Will Overwhelm Your Delivery Pipeline (opens in a new tab), January 2026).

Vendor self-reports sit far above the measured numbers and should be read as such: GitHub claims Copilot users are “up to 55% more productive” with “up to 75% higher” job satisfaction and cites a customer, Grupo Boticário, at 94% (GitHub, vendor claims, GitHub Copilot (opens in a new tab)), against the 10-15% PR-throughput gains the DX longitudinal study measured. The gains also carry new costs that token bills do not capture: A Thoughtworks report flags DORA-reported stability getting worse, changeability decay as changes sprawl across more files than they should, and rising cognitive load and burnout from supervising fast agents (A year of AI coding (opens in a new tab), June 2026). Where AI reliably wins is narrow tasks with specific scope, prototyping against unfamiliar APIs, test and boilerplate generation, and debugging; where it reliably fails is total automation, architectural decisions, and core business logic (Rotational Labs - What We’ve Learned From 3 Months Living with AI Coding Tools (opens in a new tab)).

Resources

Runtime context

  • Hud (opens in a new tab) - Runtime code sensor that feeds function-level production errors and performance into the IDE and coding agents.