Deterministic quality gates

How feedback loops from language servers through CI constrain agent behavior.

The feedback-latency spectrum and deterministic gates

Unit tests are one loop of several. The organizing idea is to surround the agent with overlapping feedback loops and order them by latency, while judging each on two axes rather than one: how fast it returns a verdict the agent can act on, and how much that verdict proves. The two pull against each other. CI is the slowest loop and proves the most; a linter is near-instant and proves the least. Ordering by latency tells you when each loop fires, not which one matters more, so the fast end is not the valuable end. This is the Fast Feedback principle applied to an agent: the sooner a fault surfaces the cheaper it is to fix, and an agent fixes a fault it sees in milliseconds far more cheaply than one a human catches days later in review.

From fastest verdict to slowest: the language server flags type errors and wrong signatures the instant the agent types, before any compile; the compiler catches what only a full build reveals; the linter checks style and simple correctness on save; the test suite is the behavioral loop, seconds per run; and CI runs the full suite plus integration and security scans as the last gate on push. Each loop catches a different class of fault, and each one caught earlier is one fewer that reaches the next.

Strict, static type systems (TypeScript, Rust, Go, or typed Python via mypy or pyright) are a strong default because they give the agent a check that fires before the code runs: the type checker is a tireless reviewer that reports a hallucinated property or an unhandled null on the exact line, and an agent wired to read that diagnostic fixes it before a human sees it. Un-typed Python is an open invitation to hallucinate data structures, so pin the shape with a TypedDict (checked statically) or a Pydantic model (validated at runtime). Linters and formatters bounce the agent off unyielding walls until its output turns modular and better-named, the same mechanism described above.

An instruction to run the linter lives in the prompt, and a non-deterministic agent will sometimes forget it, skip it under long context, or decide the code looks fine; an instruction the agent can choose to ignore is not a guarantee. The fix is to bind the check to an event so it fires regardless of what the agent decides. Two layers do this. Git hooks (pre-commit, pre-push) gate the commit itself and are tool-agnostic, running the formatter, linter, and fast tests no matter who or what made the change, but only if they are versioned into a tracked directory and wired through core.hooksPath or a framework, since .git/hooks does not travel with the repository. Agent hooks (pre-tool-use, post-edit) live in the agent’s own setup and gate its actions mid-edit, earlier still.

Both move the check off the agent’s discretion, which is the point, though a determined caller can still bypass a git hook with --no-verify. Closing that last hole means locking the gates against the agent that is trying to pass them: forbid edits to the lint and coverage configuration, protect the main branch, and block the --no-verify escape, because whatever rule you set the agent will look for a way around (Code Quality Foundations for AI-assisted Codebases (opens in a new tab), 2026). Thoughtworks tracks the same move under the name feedback sensors for coding agents: deterministic gates (compiler, linter, structural tests) wired into the agentic loop so a failure triggers self-correction, run during the session and reported clean before a commit rather than caught post-commit, implemented either as a reviewer agent or as a companion process the agent can query (Thoughtworks Technology Radar Vol 34, April 2026 (opens in a new tab)).

The pipeline write-up turns this into a self-reinforcing loop by having the agent commit its own work, so it watches the full lint-and-test suite run on every commit and must fix whatever goes red before it can proceed, sometimes grinding through dozens of cycles until everything passes; a hard 90% coverage rule forces it to write the tests it skipped, and because the agent will try to fake its way past a gate (unit tests that secretly hit real APIs, time.sleep() smuggled into async tests) the write-up adds custom lint plugins that detect those specific cheats, which the agent itself writes (Self-Reinforcing Code Quality Pipeline (opens in a new tab), 2025).

Agents also change which gates a team can afford. A practitioner interview describes two checks known since the early 2000s and shelved as too laborious. CRAP scores each function by combining its test coverage with its cyclomatic complexity. Mutation testing flips operators in the source, reruns the whole suite per flip and counts every mutant the suite fails to kill. Around 2000 the mutation run took a night, a four-minute suite repeated several hundred times, and fixing every flagged function by hand did not pay. An agent told to change the code until the tool reports clean does both inside its loop, the mutation pass in about 30 minutes. The motivation was messy code: left uncleaned, the agent began breaking one thing while fixing another until it circled. The price is throughput. Each added gate slows the agent, and the stated limit is the point where it becomes slower than a human. The interview reports still running two to four times faster with the gates in place and has not found that limit yet. This is one practitioner’s first-person account with no external measurement (LIVE: Uncle Bob on Software Fundamentals in the Age of AI (opens in a new tab), August 2026, practitioner).

One caveat holds across every gate: the diagnostic only helps if it points at the real cause. In an ABAP benchmark, Claude Haiku 4.5 failed systematically not because it could not fix the bug but because misleading parser error messages sent it chasing the wrong fault across multiple feedback rounds, so a fast loop with a lying error message is worse than no loop (Benchmarking LLMs for ABAP (opens in a new tab), early 2026).

A true message can still mislead when it names only the metric. A pre-registered study gave Claude Haiku 4.5 and Claude Sonnet 4.6 18 real Python functions, each with one known smell, either excess complexity or an exception handler that swallows errors. A deterministic judge scored five trials per function and condition and agreed with blind human labels in 90 percent of cases. Handed only the bare linter message, Haiku 4.5 fixed the smell in 28.9 percent of trials and Sonnet 4.6 in 5.6 percent. A generic request to improve the code reached 25.6 percent for Haiku and 34.4 percent for Sonnet, and a prompt explaining the smell and how to refactor it reached 83.3 percent on both. The stronger model did worst on the bare metric, the pattern expected when a model satisfies the rule instead of fixing the code. In 79 of its 90 bare-metric trials Sonnet 4.6 made the check pass without fixing the smell, for complexity by splitting the same branching across helper functions. The study names its own limits: two models, two smells, one temperature, a judge that cannot run code, and a refactoring prompt for the exception smell that states the judge’s own criterion (prompt-vs-metric-eval (opens in a new tab), September 2026).

Coaching guides on linter findings

Habit Hooks applies the same idea as a tool. It runs the linters a project already has and prints a short coaching guide in place of each raw violation (habit-hooks - Habit Hooks (opens in a new tab), MIT, version 1.5.0 as of September 2026). Its maintainers argue that a long rule document in the context window makes an agent worse, while a guide delivered with the finding arrives when it applies. That argument is the project’s own claim. The repository dates from May 2026 and predates the study above, which tested prompt wording on single functions and did not measure a full run of the tool.

The design separates detection from advice. Two commands run on a pipe: the first runs the detectors and emits findings as JSON, the second groups them by smell, prints the guide and sets the exit code. Each sensor translates a tool’s rule ID into a tool-independent smell key, so ESLint’s max-params and Ruff’s PLR0913 both become too-many-parameters. The guide is chosen by smell and never by the reporting tool, so one guide serves every language whose linter detects the smell (habit-hooks - Habit Hooks architecture (opens in a new tab)). Plugins cover Python, TypeScript, PHP, Java and Ruby. A project replaces a single guide by placing a file of the same name in its own configuration directory, and an upgrade leaves that file alone (habit-hooks - Habit Hooks (opens in a new tab)).

The shipped guides share a shape a team can copy into its own linter messages. Each names the cause behind the number, lists the moves in order, names the shortcut that only moves the counter, and ends on a test the reader can apply. The complexity guide tells the agent to lift precondition checks into early returns and to turn a chain switching on one value into a lookup table. It then rules out merging conditions or rewriting branches as ternaries to lower the score (habit-hooks - high-complexity guide (opens in a new tab)). The parameter guide treats an options object named after its function as the same list renamed, and asks for the domain entity the parameters belong to (habit-hooks - too-many-parameters guide (opens in a new tab)).

The project’s authoring skill says when a guide is worth writing: only where the agent’s default fix is shallow or wrong. A rule whose own message already produces the right fix stays uncoached. A guide states the risk, the move and the expected outcome, and leaves the observation to the linter. The review checklist caps a guide for an architectural smell at 150 words, allows 30 to 60 for code hygiene, and forbids suggesting a suppression comment (habit-hooks - skills (opens in a new tab), as of October 2026).

Adoption on an existing codebase runs through a baseline, which the tool calls snoozing. One command records today’s findings in a checked-in index, and later runs report only new smells. The index stores the content that was approved, so editing a baselined file brings its findings back while the agent is working in that file. Structural smells such as an oversized function fail the run by default. Judgement calls such as duplicated code or a swallowed exception coach without failing it. The recommended snippet for the agent’s instruction file forbids baselining without the user’s approval, which closes the same hole as blocking --no-verify (habit-hooks - Habit Hooks (opens in a new tab)).

The tool also separates “found nothing” from “ran nothing”. Exit code 0 means clean, 1 an enforced finding, and 2 that the tool itself failed, for example on a bad configuration key or a configured plugin that is not installed. An empty pipe between the two stages is reported as an incomplete run and exits 2. A finding with no guide is printed uncoached instead of being dropped (habit-hooks - Habit Hooks (opens in a new tab)). A clean run still proves little about behavior: the pass message says the tool catches structural smells, not correctness or design, and tells the agent to get a reviewer sub-agent before declaring done. That handoff is on Creator and verifier roles.

Formal specification as the deepest gate

The ladder above ends at CI, which still leaves the fault class no linter, type checker, or test suite reliably reaches: the interleaving bug in concurrent and distributed code. One open-source Python harness attacks exactly that gap by putting the coding agent in front of a model checker instead of a test runner. The agent reads the target repository, writes a TLA+ specification of the implementation together with the invariants that state its correctness properties, model-checks it, and then works backward from any violation to reproduce it against the real code (Specula (opens in a new tab), Apache-2.0, active as of August 2026). The verdict is a different kind than a green test run: a counterexample trace proves that a reachable sequence of steps breaks a stated property, rather than showing that the cases someone thought of still pass.

Two design choices are worth copying independent of TLA+. The work is decomposed into a fixed chain of agent skills, from code analysis through spec generation, harness generation, and validation to bug confirmation, where each step writes files the next one consumes, so a run is inspectable at every handoff and any single stage can be re-run alone against an existing spec. And the checker reaches the agent as MCP tools (a trace debugger, a spec analyzer, an invariant checker) rather than wrapping the agent, which keeps the deterministic verdict on the agent’s own tool surface. The cost sets the frequency: the toolchain wants Java and a model checker beside Python, 32 GB of RAM and 100 GB of disk, and Linux or WSL2, so this is a campaign run against something like a BFT consensus implementation, not a per-commit gate. One operational wrinkle generalizes past this tool: the project reports that model providers may block the bug-reproduction phase unless the account holds trusted-access approval, and ships a hybrid configuration that routes just that phase to another model, so safety filtering rather than capability can be the binding constraint on agentic security work (specula-org - Specula (opens in a new tab)).

Compilation as a control boundary

A Markdown workflow separates the agent-facing instruction from the executable automation. YAML frontmatter declares triggers, permissions, and tools; the prose body describes the task. A compiler then turns that source into a lockfile containing the hardened workflow, making compilation a reviewable boundary where configuration can be validated before execution (GitHub - How They Work (opens in a new tab)).

Permission-separated outputs

The agent job runs with read-only permissions. Requested writes are buffered as artifacts, passed through deterministic filters and threat analysis, and only then executed by separate jobs with narrowly scoped write permissions. This turns an agent’s proposed side effect into an inspectable intermediate result rather than granting the model direct authority (GitHub - Security Architecture (opens in a new tab)). The same design suggests a general rule for agentic automation: keep reasoning and side effects in different stages, and make the transition explicit in the workflow.

Defense in depth

The control surface extends across compile-time schema and expression validation, SHA-pinned actions, runtime isolation, network allowlists, MCP server sandboxing, tool allowlists, and output sanitization. These layers do not make a compromised agent trustworthy; they constrain what it can reach and what it can externalize, reducing blast radius when one layer fails (GitHub - Security Architecture (opens in a new tab)).

How these gates line up as pipeline stages with time budgets, from pre-commit to canary rollback, is on Agentic continuous delivery.

Resources

Agentic workflows

• Agentic Workflows (opens in a new tab) - Active MIT-licensed framework for compiling Markdown agent instructions into guarded CI workflows with isolated writes, tool allowlists, and network controls.

Agent-readable diagnostics

• Habit Hooks (opens in a new tab) - MIT-licensed Python CLI that maps linter rule IDs to shared smell keys and prints one coaching guide per smell for coding agents.
• prompt-vs-metric-eval (opens in a new tab) - Pre-registered MIT-licensed study comparing bare linter messages with refactoring guidance on 18 Python functions across two models.

Formal verification

• Specula (opens in a new tab) - Apache-2.0 Python harness where coding agents write TLA+ specs and invariants for a codebase, model-check them, and reproduce violations at code level.