Skip to content

Knowledge base

Evaluating non-deterministic systems

How to test output that varies between runs, from unit tests to pass-rate baselines, and what a public eval leaderboard does and does not prove.

Written by

Updated:

Agentic Engineering 5

Our position. We make an evaluation explain its verdict, because the explanation is the signal (opens in a new tab) and a bare score is not. A new model version does not inherit the trust of the old harness: it earns its own (opens in a new tab).

The graders need the same care as the agent. Layered checks run cheapest first, each grader is calibrated against known good and bad output, and a held-out slice keeps an optimization loop from grading its own fix.

How do you evaluate output that varies between runs?

Tests and gates verify the code an agent writes. A parallel discipline verifies the agent system itself, and it cannot use the same tools because the object under test is non-deterministic. The same input yields different outputs, and outputs are sensitive to small prompt changes, so a single pass-or-fail run proves nothing. The working method is statistical: decompose the agent into deterministic parts (tool calls, calculations, structured-data handling) that take ordinary unit tests, and non-deterministic parts that need evaluation over a distribution. Treat every change, a prompt tweak, a model upgrade, a parameter shift, as a statistical experiment: collect a baseline over many runs, introduce the change, and compare distributions with confidence intervals rather than asserting a single result. For outputs too open-ended to match exactly, use an LLM as a classifier against explicit criteria to turn free text into pass/fail signal, with majority voting on the ambiguous cases (How to test AI agents (opens in a new tab)).

The eval suite is itself a thing you have to get right, and a common way it lies is by asserting implementation details or testing capabilities already in the training data. Vercel hardened its own suite by removing test leakage, resolving contradictions, shifting to behavior-based assertions, and adding tasks that target APIs the model has not seen, then running every configuration with retries to rule out model variance; only after that did their measured differences mean anything (Vercel – AGENTS.md outperforms skills in our agent evals (opens in a new tab), 27 January 2026).

The continuous delivery community grades a coding agent in three layers, cheapest first. Deterministic graders check structure, arithmetic and file references in milliseconds with almost no false positives. Transcript graders read the sequence of tool calls and confirm that the agent gathered evidence before it reported a finding. One example threshold requires evidence actions to make up at least 40% of all actions. LLM rubrics come last and judge what the other two cannot, such as accuracy, severity ratings and invented findings, at seconds and tokens per case (MinimumCD.org and DojoConsortium.org - AI Eval Methodology for Coding Tools (opens in a new tab), as of March 2026). The first two layers cover 60 to 70% of what a team needs in that guide’s estimate. So pre-commit runs only deterministic graders, every push adds transcript graders, a pull request adds the LLM rubrics, and a release branch runs the full suite over several trials (MinimumCD.org and DojoConsortium.org - Team AI Evals for Coding Tools (opens in a new tab), as of March 2026).

Two pass rates separate capability from reliability. pass@k is the probability that at least one of k runs passes, the capability ceiling. pass^k is the probability that all k runs pass, the reliability floor that unattended use depends on. The guide’s reference targets are pass@1 above 80%, pass@5 above 95% and pass^5 above 60% (MinimumCD.org and DojoConsortium.org - AI Eval Methodology for Coding Tools (opens in a new tab), as of March 2026). Every capability suite gets a twin negative suite of clean inputs that must produce no finding. The negative suite gates a release at pass@1 of 90% or more, while capability pass rates stay ungated during early development (MinimumCD.org and DojoConsortium.org - Team AI Evals for Coding Tools (opens in a new tab), as of March 2026). A platform team serving many teams reads pass@5 holding above 95% as saturation: the suite no longer discriminates and needs harder cases. Its shared fixture portfolio therefore spans several deliberately flawed repositories plus one clean repository (MinimumCD.org and DojoConsortium.org - AI Evals for AI Enablement Platforms (opens in a new tab), as of March 2026).

The graders need calibration of their own. Each grader must pass a reference solution and fail known-bad output, and every disagreement with a human verdict gets investigated (MinimumCD.org and DojoConsortium.org - Team AI Evals for Coding Tools (opens in a new tab), as of March 2026). As reported by the MinimumCD page, fixing grader bugs in the CORE-Bench research benchmark raised measured agent performance from 42% to 95%. A task is specified tightly enough when two domain experts would give the same verdict on the same output (MinimumCD.org and DojoConsortium.org - AI Eval Methodology for Coding Tools (opens in a new tab), as of March 2026).

Calibration changes where the product decision is still open. A web book on delegation works through two dashboard variants that each implement their own priority rule correctly and pass every code check. Choosing between them is a product judgment the team has not made yet, so writing the preferred answer into a test would encode an unmade decision. The reviewer, human or agent, is graded on its reasoning instead: whether its findings follow from the evidence, whether it names material consequences, and whether it recognizes when it needs more information. A reviewer that always favors one kind of case misses the judgment the task requires, while different recommendations pass when their reasoning is sound and they respect the agreed constraints. Where a clear acceptance decision exists, count incorrect approvals and incorrect rejections separately, assess the reviewer on cases kept apart from those used to refine its criteria, and recheck it whenever its model, instructions or criteria change (The Elastic Loop: Grading (opens in a new tab)). Once the team settles the rule, deterministic tests enforce it as quick feedback, and whether the rule still serves the product stays with review and product learning (The Elastic Loop: Grading (opens in a new tab)).

A model upgrade runs against a recorded baseline. The negative suites run first, because new false positives are the likeliest regression. Then pass@k and pass^k are compared side by side, and the transcripts are read for new failure patterns. An improved aggregate can hide single tasks that got worse, so the comparison goes task by task. In production, a failed user interaction becomes a new eval case tagged as a production failure and is reproduced on a minimal fixture before the prompt changes. Alongside that, the team reads five to ten production transcripts by hand every week (MinimumCD.org and DojoConsortium.org - Team AI Evals for Coding Tools (opens in a new tab), as of March 2026).

In production, evals and observability turn out to be the same problem, evals being the batch case where you know the inputs ahead of time and observability the real-time case where you do not, so agent traces (large, deeply nested, full of unstructured text) become the substrate for both; the same LLM-as-judge that scores a batch dataset runs as an online eval over live traces, and the whole loop of building datasets, defining correctness and hallucination metrics, and hill-climbing without regressing is the agent’s version of a test suite (LangChain – Interrupt 2026 keynote on the agent development lifecycle (opens in a new tab), 19 May 2026).

The bootstrap that scales human judgment: have a domain expert grade traces and justify each grade, then run an LLM over those justifications to distill reusable scoring functions, which is how the best teams put clinicians, lawyers, and other non-engineers directly into the loop (Braintrust – Agent observability vs traditional observability (opens in a new tab), 28 May 2026). An Anthropic research PM describes the same craft from inside model development: an eval is only useful when it matches the shape of a real user task rather than a leaderboard like Terminal Bench that can be gamed, a sample as small as dozens of cases can be enough to expose a defect and give the team something to hill-climb against, and the qualities that resist quantification, a model’s character and personality, get evaluated by reading hundreds of transcripts to build intuition and by having the model grade its own outputs against explicit criteria (Anthropic research PM on evals and model character (opens in a new tab), 17 May 2026).

The tooling side of this is starting to ship as an agent in its own right: vendor eval platforms now embed an AI assistant that turns log patterns into scorers and datasets and optimizes prompts from human annotations through natural language (Braintrust’s Loop is one vendor example).

That tooling is extending past scoring into change-making, closing an optimization loop around the eval suite. One platform’s stated design derives eval datasets and a ranked root cause from production traces, has a change writer propose a single scoped fix to a prompt, tool definition, or the surrounding code, then scores the candidate against an evaluator built and calibrated separately from the writer; the candidate is scored against the whole suite rather than only the failing case, past failures are retained as regression tests, and a candidate that does not move the score on a held-out slice of traces is discarded rather than shipped, so only a change that beats baseline on traces it never saw reaches a human as a pull request (Optimize agent (opens in a new tab), vendor description, as of July 2026). The separation is the load-bearing part. The machinery that writes the fix would happily grade it, and an evaluator derived from the same diagnosis it is meant to check measures nothing, so the held-out slice is what stops the loop from hill-climbing on its own explanation.

The deeper lesson is that a dataset of good inputs and outputs is not a specification. Specifying what an agent should do means capturing the rules it must never break (never discount more than 10%), the ontologies and internal terminology it operates in, the rights and roles that change its behavior, and explicit robustness requirements (how many typos or rephrasings before it fails), all authored independently of the implementation so the tests survive a change of framework and can be re-pointed at security and penetration testing. A bigger model is not automatically a better or safer agent: broad capability widens the attack surface, and some jailbreaks work better on larger models precisely because they understand the trick; what you want is a model good enough to perform but constrained enough that it cannot do arbitrary harm (Safe Intelligence – Spec-driven validation of agents (opens in a new tab), 31 May 2026).

A conference talk turns that same ontology into a runtime check rather than prose inside a specification. After a tool call returns, its result passes through a formal reasoner before the agent may act on it, so business invariants a shape validator cannot express become enforceable: disjoint classes catch a payout routed to the wrong kind of party, a single-valued property catches a second refund against one order, an enumerated range catches a status the model invented. The distinction worth carrying is between validating the shape of a tool’s output and validating whether it is allowed to be true (Why Agentic Systems Need Ontologies (opens in a new tab), 22 July 2026, practitioner).

An open-source agent product shows how to test the code around a model without testing the model. Several of the more than 200 smoke scripts in exxperts spawn the real server against a synthetic OpenAI-compatible gateway on a local port. They assert what the gateway received, for example that a per-turn stanza in the system prompt lists the room’s tools, carries the memory-recall line word for word and stays at 400 characters or less (EXXETA - room tools stanza smoke (opens in a new tab), version 0.14.1, October 2026). Where a published number depends on a prompt, the prompt is pinned. One smoke renders the memory-summary prompt for a fixed conversation at a fixed clock and compares its sha256 with the hash taken before a refactor. The reason given is that this prompt is what the project’s 43-of-50 memory result was measured on (EXXETA - Remember single-pass pin smoke (opens in a new tab)).

The same project’s memory bench grades the memory pipeline against a fixture in which every planted fact has a reference code, so a score names the fact that went missing. Scripted models calibrate the bench. An exact model must score every category, and anything less counts as a bug in the bench. A lazy model that appends every line and never supersedes must fail visibly, and it scores 1 of 33 on the default fixture. A class of no-hint questions shares no content word with the planted sentence beyond the named entity, and the fixture’s self-check rejects a build where one does. The room then has to decide to search on its own. Phrase lists turned out to mis-score abstentions: 11 of 20 honest “I don’t know” answers were marked wrong for their wording, while the LongMemEval judge accepted all 20. The bench now gates on whether an answer invents a fact and reads the phrase list only as evidence (EXXETA - memory bench README (opens in a new tab), project self-claims, as of October 2026).

What does an agent topping a public eval prove?

OpenAI’s Parameter Golf challenge drew more than 1,000 participants and over 2,000 submissions under a 16 MB artifact cap and a 10-minute training budget on 8 H100s, and OpenAI independently reproduced and verified the record-track submissions (OpenAI - What Parameter Golf taught us (opens in a new tab)). Weco’s CEO and co-founder presented the company’s entry, Aiden, a multi-agent prototype that read papers and pull requests, ran experiments, applied a quality gate, and published pull requests; the Weco presentation reports 47 accepted record entries across the challenge, seven of them Aiden’s, roughly 1,300 internal experiments over 22 days, and a single-H100 execution setup (An AI agent became the #1 contributor in OpenAI’s hiring challenge (opens in a new tab), 16 July 2026). Those figures are Weco’s own; OpenAI’s retrospective does not mention Aiden or independently confirm them.

The case is useful because a submission’s local score was not the only quality signal the challenge rewarded. Evaluating an agent like this means reading four distinct axes: search throughput (the roughly 1,300 experiments above), validity at an external quality gate (records that pass OpenAI’s own verification under the 16 MB artifact cap), compute efficiency (results per H100-hour), and downstream reuse or influence (whether other participants build on the work). Accepted pull requests, reuse of a submission’s ideas, and citation lineage measure only that last axis. Weco reports a PR h-index of 10 for Aiden against 7 for the next contributor, but that h-index is a practitioner-defined influence proxy and Weco’s own analysis, not proof of quality (Weco presentation, 16 July 2026 (opens in a new tab)). The strongest public evidence is a single merged submission: pull request #1736 documents a 1.06549 bits-per-byte result and a lineage that draws on CaseOps, gated attention, QuantGate, and other community work (openai - openai/parameter-golf#1736 (opens in a new tab)).

Two lessons carry over to evaluating any non-deterministic system. First, the agent’s strength was execution, search, implementation, and recombination; most of the productive primitives came from human papers and participants, and the competition and eval design stayed human leverage. Second, the conceptual framing: the eval is the loss function and the data, while the codebase abstraction is the architecture that biases the search. The Weco presentation also offers an illustrative example of the eval boundary, claiming that a loose scoring API allowed train/test leakage while a strict API closed that path; no public dataset or code accompanies the example, so treat it as a speaker illustration rather than a measured result (Weco presentation, 16 July 2026 (opens in a new tab)).

Resources

Agent optimization

Autonomous research

Eval methodology