Skip to content

Knowledge base

Creator and verifier roles

Patterns that separate making from checking, why self-grading fails, and how to wire a verifier into the loop, for teams running more than one agent.

Written by

Updated:

Software Factories 5

Our position. We separate building from checking: the checking station must not report to the building station, and review runs on specialized agents on different models, so a second vendor’s model reviews the first one’s code (Separate building from checking (opens in a new tab)). A reviewer kept as its own sub-agent reviews with a clean pair of eyes instead of inheriting the bias of the work it checks (Skills versus subagents (opens in a new tab)).

A separate agent is not enough on its own. In a measured loop, a judge that read only the artifact and the diff still accepted regressions, so independence has to reach the real-world signal.

Which patterns separate making from checking?

Self-grading

Anti-pattern.

Context. An agent finishes a change and is asked to judge it, or a loop gate trusts the agent’s own report of progress.

Problem. The agent praises mediocre work and talks itself out of the real issues it flagged. In a measured loop it claimed an improvement in every cycle while most cycles measured zero or worse.

Way out. Hand the judgment to a separate verifier that grades against a validation contract and reaches the real-world signal.

Watch for. A loop gate that reads the agent’s own progress claims. In the measured loop, a gate that trusted them accepted later regressions until the best state reached had eroded by 19 percent.

Related. Separate verifier, Validation contract.

Separate verifier

Pattern.

Context. A generator agent produces code or a product in a loop and something has to decide whether each round is good enough.

Problem. A review that shares the generator’s context inherits its reasoning, and a judge that reads only the artifact and the diff still accepts regressions.

Therefore. Run the verifier in fresh context and wire it into the loop as an evaluator. Let it drive the output the way a user would, mix deterministic checks with model judgment, and run it on a different model provider where you can. A deterministic gate can spawn the reviewer once its own checks pass.

Watch for. Validation costs wall-clock time, because the verifier waits for real execution. A reviewer tends to pad its findings to look thorough, so accept a short pass and cap nits. A managed grader may keep its reasoning hidden from the caller.

Related. Self-grading, Validation contract, Production evidence.

Validation contract

Pattern.

Context. A feature or mission is being planned and no code exists yet.

Problem. Tests written after implementation confirm decisions instead of catching bugs, and a generator can move the goal posts halfway through.

Therefore. Write the definition of done during planning, independent of the implementation: assertions mapped to features, or a rubric of gradeable criteria derived from a known-good artifact.

Watch for. Every criterion has to be gradable. Ask whether the output follows stated design principles, not whether it is beautiful.

Related. Separate verifier, Self-grading.

Why can’t an agent grade its own work?

The single most reliable finding across the sources is that an agent cannot grade its own work. Ask it to and “it’s likely going to praise it,” even when quality is “obviously mediocre to a human observer”; out of the box a model is “a poor QA agent,” repeatedly flagging real issues and then “talking itself into deciding they weren’t a big deal” and approving anyway (The AI Automators – harness design for long-running agents (opens in a new tab), 25 March 2026). The fix is a separate verifier run in fresh context so it “is not biased by the implementation agent’s reasoning” (production agent loops (opens in a new tab)); single-agent harnesses reach the same end by branching into a clean review context before bringing fixes back (Pi review workflow (opens in a new tab), 31 January 2026).

A controlled pilot measures the failure in a running loop and shows that a separate judge is not enough on its own. Two independent researchers held the agent and its tools fixed and varied only what the gating evaluator could see, with a simulated user outside the agent’s reach measuring real outcomes. The agent claimed an improvement in all 54 cycles, yet 56 percent of them measured zero or worse. A gate that trusted those claims accepted later regressions until the best state it had reached eroded by 19 percent. A strong in-band judge reading the artifact, the diff and its own verdict history still accepted cycles of which 44 percent were regressions, and rejected 38 percent of real improvements. The gap closed only on a task whose success could be read from the artifact itself, so the authors conclude that the judge has to reach the real-world signal it grades. The paper is a preliminary draft at pilot scale with one agent (When Do Agent Loops Mistake Stagnation for Progress? (opens in a new tab), July 2026, revised September 2026).

A benchmark isolates one mechanism behind self-grading. Self-Correction Bench injects the same error into a conversation either as the user’s or as the model’s own, with everything else identical. Across 14 open-source non-reasoning models it measures a 64.5% blind spot: the models correct the error when someone else made it and miss the identical error in their own output, so the capability exists and is not activated. The authors trace the gap to fine-tuning data that lacks error-correction sequences, and appending the word “Wait” cut the blind spot by 89.3% without any training (Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models (opens in a new tab), July 2025, revised August 2026). The finding supports presenting work to a reviewer as another party’s output, and a web book on delegation draws that conclusion for now, using independent review where the extra mistakes it catches justify its cost (The Elastic Loop: Harness (opens in a new tab)).

The account states the principle plainly: “agents produce better code when they can gauge the quality of the code they produce themselves rather than relying on us to check it” (humans and agents in the loop (opens in a new tab), 4 March 2026).

How do you wire a verifier into the loop?

Wiring the evaluator into the loop is the production move. The AI Automators model it on a GAN, a generator agent paired with an evaluator agent, the difference from ad-hoc “LLM as a judge” being “an evaluator wired into a production loop.” Three requirements make one work: make subjective quality gradable (ask “does this follow our principles for good design?” not “is this beautiful?”), weight the criteria toward the model’s demonstrated strengths, and let the evaluator interact with the output, for example driving the app through a browser MCP and screenshotting it like a real user. Define done before building: Factory writes a validation contract during planning, “before any code,” that “defines correctness independently of implementation,” sometimes hundreds of assertions each mapped to a feature, because “tests written after implementation don’t catch bugs, they confirm decisions” (Factory – Missions (opens in a new tab), 6 May 2026).

Anthropic’s harness team independently negotiates the “definition of done up front” so the generator “couldn’t move the goal posts halfway through.” Factory then runs two validator types: a scrutiny validator (tests, types, lint, per-feature code review) and a user-testing validator that spawns the app and drives it through computer use, where “most of the wall-clock time is spent waiting for real-world execution instead of generating tokens.”

Two further mechanics sharpen verification. Put the right model in each role: Factory’s “droid whispering” holds that “no single model nor model provider is best at all three” of planning, implementation, and validation, and deliberately runs validation “on a different model provider entirely” so it “is not biased by the same training data,” making model-agnostic design a structural edge. And grade with a mix of check types: OpenAI’s retraining example runs deterministic graders (name match, length, cosine similarity) alongside an LLM-as-judge, “mixing deterministic checks with semantic judgment,” and versions the agent’s prompt so a regression rolls back to a known-good version (OpenAI Cookbook - self-evolving agents (opens in a new tab)).

Anthropic’s Managed Agents turn the separate grader into a platform feature. The caller defines an outcome with a markdown rubric of explicit, gradeable criteria, and a grader in its own context window scores the agent’s work against each criterion and explains every verdict. The agent then revises until the grader reports the outcome satisfied or the run hits max_iterations, which defaults to 3 and allows at most 20. Each round reports a result: satisfied, needs_revision, max_iterations_reached, failed when the rubric does not fit the task, and interrupted. The documentation recommends deriving a rubric from a known-good artifact. The grader’s reasoning stays hidden from the caller, and the feature is a beta API (Anthropic - Define outcomes (opens in a new tab)).

A deterministic gate can trigger the verifier itself. Habit Hooks, a linter-coaching tool, ends every clean run by telling the implementing agent to spawn a reviewer sub-agent, and ships the brief for it. The implementer is told not to review the change first, because the fresh read is what the reviewer adds. The brief excludes everything the gate already checks and lists what the gate cannot see: correctness, design, test gaps and edge cases. It also counters a reviewer’s urge to look thorough. A two-line pass is a valid result, every finding cites file and line, and nits are capped at two. When the reviewer asks for changes, the implementing agent takes the findings to the user and agrees which to act on before it fixes any (habit-hooks - habit-hooks-review skill (opens in a new tab), as of October 2026).

Resources

Verification tooling

Research