Skip to content

Knowledge base

Supervision and drift

Supervising coding agents takes verification and review capacity that grows with agent output, and the coding skills it requires erode when developers lean on AI. Teams hold quality with deterministic checks in CI, inferential reviews outside it, named human reviewers who own the merge, and continued practice of the craft.

Written by

Updated:

Tools 4

Our position. We hold that a team’s engineering knowledge needs to live in the system, not in people’s heads (opens in a new tab): linters, structural tests, acceptance criteria and deterministic guardrails, rather than reading every line. Skipping understanding to keep up with generation speed (opens in a new tab) costs the ability to recognize problems. The less a team supervises, the more of its feedback has to be deterministic (opens in a new tab).

How do you verify and review agent-written code?

When agents write most of the code, verification becomes the load-bearing wall. The State of Play talk offers a mental model of feed-forward guides that steer an agent before it acts and feedback sensors that catch problems after, each either inferential (an LLM judging an LLM, running on a GPU) or computational (static analysis, linters, codemods, running on a CPU). Its sharpest placement rule is to keep inferential sensors out of CI, because “you don’t want the green or red state of your pipeline to depend on semantic interpretation of an LLM”, and to prefer a cheap deterministic lint rule over stuffing context on every run for a mistake the agent makes once a week.

Expensive inferential reviews for modularity, dependency freshness, and security belong in a weekly “garbage collection” pass, triaged like security vulnerabilities rather than run per commit; pairing codemods such as OpenRewrite with AI is the pattern behind Amazon’s claimed hundreds of developer-years saved on Java upgrades (State of Play: AI Coding Assistants (opens in a new tab), June 2026). Baz builds its own version of this with two complementary retrieval mechanisms, vector similarity plus a deterministic module and reference graph over the code, and benchmarks its output with an LLM-as-judge deliberately run on a different model than production because “models like their own results” (Baz on building a code-review agent (opens in a new tab), June 2025).

Review is where the parallelism actually bottlenecks. As PR volume explodes, the informal pattern of posting to a team channel and letting one responsive person absorb most reviews collapses, so teams have to assign reviews to individuals, enforce service-level objectives, and optimize per-response latency rather than total timeline, since review is a quality gate; the only known way to train good reviewers is apprenticeship (Developer Experience in the Age of AI Coding Agents (opens in a new tab), December 2025). One practitioner argues code review is now the single most important engineering skill and that teams should interview for it instead of algorithm puzzles (Vibes won’t cut it (opens in a new tab), August 2025). Sourcegraph’s Amp attacks the review load with a custom diff viewer that gives a guided “tour of the change” naming which files to read first, plus shareable threads so teammates learn prompting from each other (Sourcegraph on Amp (opens in a new tab), December 2025).

Ona (formerly Gitpod) redesigned its agent workspace on the same premise, that every switch from a chat window to a code editor forces the brain to flush and reload context, so it pulled review into the conversation with a searchable file tree, readable diffs, and inline comments that feed straight back into the dialogue, and decoupled the conversation from the runtime environment so a developer can step away and return to find the agent where they left it (Ona, vendor, Redesigning Ona conversations (opens in a new tab), April 2026). Accountability matters as much as tooling: OpenHands found that PRs owned by the bot were an anti-pattern, letting the triggering human self-approve and leaving failing PRs orphaned, so the fix is to put the human’s face on the PR and make them own the merge and any downstream breakage.

The counterpart mindset is that agent-written code is cheap and disposable; OpenHands reports discarding roughly half of its speculative, voice-dictated PRs, iterating in-thread when an agent is close and restarting fresh when it is far off (OpenHands on coding agents (opens in a new tab), July 2025).

Does relying on AI erode the skills to supervise it?

A recurring practitioner warning qualifies that answer: leaning on AI erodes the very skills needed to supervise it. One practitioner removed all LLM integrations from their editor after noticing a loss of what they call Fingerspitzengefühl, the low-level intuition a senior develops for pointers, standard-library choices, and “something feels off”; they now keep AI fully separate from the editor, adds context manually to keep the friction deliberately high, insists that security-critical code such as JWT signing and RBAC be written and reviewed by humans, and concludes that “if you cannot code without AI, you cannot code” (Why I stopped using AI code editors (opens in a new tab), April 2025).

Anthropic’s own research frames this as a paradox of supervision: “effectively using Claude requires supervision, and supervising Claude requires the very coding skills that may atrophy from overusing AI”, with atrophy reported within months, which is why one engineering director told his team not to use agents for tasks requiring critical thinking (We all fell for it (opens in a new tab)). The same critique rejects the idea that natural-language prompting is simply the next abstraction layer above Python the way C sat above assembly, on the ground that “a high level of ambiguity is not a higher level of abstraction”. Notably, professional software engineers are described as the last group to adopt these tools, an inversion of prior shifts like version control and cloud where developers reached for the new thing first (Vibes won’t cut it (opens in a new tab), August 2025).

Resources

Articles

Posts

Talks and videos