Skip to content

Knowledge base

Specifications, tests, and context

The specification is the source of truth and its tests check the code against it. When the two drift apart, the spec's owner decides which side changes. Documentation now has a second reader, the agent, which needs a small, ranked and maintained instruction layer instead of an exhaustive one.

Written by

Updated:

Agentic Engineering 5

Our position. We hold that the specification is the source of truth. The acceptance tests come out of it, the agent implementing the change leaves them alone, and where code or tests drift from the spec, its owner decides which side changes and records why. The default is the lightest thing that works (opens in a new tab), and spec weight is added only when the work earns it. For the agent’s instructions we merge a team’s files into one AGENTS.md and give it an owner, because the file was never the hard part (opens in a new tab).

Should the specification or the tests be the source of truth?

Where the work earns a spec, spec-driven work inverts the time budget of development. Ona’s onawalk experiment produced a working 3D application from 30 minutes of spec writing and 10 minutes of agent execution, and draws the operative lesson: a vague prompt yields a vague result, while a spec that pins down shader parameters, camera behavior, and loading strategy yields exactly that (Ona - Spec-driven design: why planning is the new coding (opens in a new tab)). Practitioner reports reach the same conclusion from the other direction: specification writing quality determines the quality of AI output, and decomposition into agent-sized pieces remains the central technical skill (AI Productivity Research Evidence).

Specifications are necessary but not sufficient. The Dark Factory patterns name “Big Design Upfront as Source of Truth” an anti-pattern: specifications are incomplete, hypothesis-based, and contain errors, and the more confidently they are formulated the more dangerous unquestioned implementation becomes. The source of truth of an autonomous software factory is programmatic tests and fitness functions; the specification describes intent, tests check reality, and where the two diverge, reality is right (Dark Factory Patterns). We keep the spec as the source of truth and take the warning seriously: a failing test sends the question to the spec’s owner, who corrects the spec where it was wrong.

The same source adds an ordering constraint for teams scaling agent output: validation and observability first, production volume second. A factory that cannot reliably distinguish good output from bad should not run at production scale (Dark Factory Patterns). For engineering teams this means investing in test suites, acceptance criteria, and fitness functions before investing in more agent throughput; verification design is itself listed among the skills that matter most going forward (AI Productivity Research Evidence).

The failure mode when specifications are pushed too hard is more confusion, not more precision: the more detailed the instructions, the more the agent overlooks, and the fix is verification and iteration rather than more prompting (Modernisierung mit AI (opens in a new tab)). The same talk names the gap autogenerated bullshit, the four-way divergence between what the requirements said, what the agent thinks it did, what the user thinks the agent did, and what actually happened; a requirement to implement user authentication (on the slide, in German: “Implementiere User Authentication”) becomes a login form with no database and no session management (Modernisierung mit AI (opens in a new tab)). This is why several practitioners treat tests as immovable. In the System Initiative acceptance-test repository a standing rule states the tests are the source of truth and must not be changed, so any failure is a regression in the binary, never a test to adjust (If humans architect the system, the AI writes the code (YouTube) (opens in a new tab)).

Ordering matters. Tests written after implementation get optimized toward the code already in the agent’s context rather than pinning intended behavior, so the test comes first (AI Labs - Three levels of AI UI design (YouTube) (opens in a new tab)). Plans can be pressure-tested the same way: the System Initiative loop runs a generating agent against a deliberately grumpy adversarial reviewer that must prove the work is secure and architecturally complete, capped at five rounds before a human arbiter is forced in, which stays cheap because a discarded session costs about ten minutes (If humans architect the system, the AI writes the code (YouTube) (opens in a new tab)). A good structural split also hands you the test pyramid for free: the Capability-Centric Architecture article notes that pure-logic essence tests run in milliseconds, realization tests mock infrastructure, and contract tests verify each capability keeps its promises (Capability-Centric Architecture (opens in a new tab)).

The continuous delivery community ranks the specification artifacts, so an agent never has to guess which one wins. Its delivery contract orders six artifacts from most to least authoritative: the human intent, the user-facing behavior as BDD scenarios, the feature description with its architectural constraints, the acceptance criteria, the organization-wide system constraints, and last the implementation. When two of them disagree, the lower one is changed. An unstated ranking counts as a failure of its own, because an agent facing a conflict then picks a side silently (MinimumCD.org and DojoConsortium.org - Agent Delivery Contract (opens in a new tab), as of September 2026). The ranking decides which document gets edited. It does not make a specification proof of correctness, which stays with the tests as above.

The feature description marks where an agent’s discretion ends. Besides hard musts, explicit prohibitions and soft preferences, it lists escalation triggers: conditions under which the agent stops and asks instead of deciding, such as a change that would reach into the authentication layer. The acceptance criteria come in two parts. A done definition states outcomes a reviewer can confirm without reading the code, and an evaluation design lists cases with known-good outputs that the pipeline checks on every commit (MinimumCD.org and DojoConsortium.org - Agent Delivery Contract (opens in a new tab), as of September 2026).

The same contract asks two different questions at two different times. The acceptance criteria ask on every commit whether the code works. The intent carries a testable hypothesis that names the change, the outcome it should cause and the reason, and production metrics answer after deployment whether the change achieved its purpose. The worked example is a rate limiter that passes every test and still leaves tail latency where it was, because abusive clients were not the cause. That refuted hypothesis becomes input for the next intent. Reading green tests as a confirmed hypothesis is the confusion the source warns against (MinimumCD.org and DojoConsortium.org - Agent Delivery Contract (opens in a new tab), as of September 2026).

How does documentation become agent context?

Documentation now has two audiences, and the agent-facing half is becoming an engineered layer of its own. One framework grades this “agentic layer” from a bare CLAUDE.md up through specs directories, sub-agents, coordination files, and closed-loop prompts, with the thesis that once the layer is powerful enough, agents operate the codebase more effectively than any individual engineer (The Codebase Singularity (opens in a new tab); see Agentic Engineer Maturity Model). Context engineering, the design of the entire model input including prompts, retrieved data, memory, and tools, is correspondingly treated as an architectural concern in its own right, with the guidance to own prompts as first-class code rather than outsourcing them to framework black boxes (AOE - AI Technology Radar: Context Engineering (opens in a new tab)).

Size discipline matters. Claude Code’s creator keeps his CLAUDE.md at roughly 2,500 tokens; the agents.md standardization effort recommends about 70 lines; and an ETH Zurich study found that context files such as AGENTS.md do not generally raise coding agents’ task success over no context file, while raising inference cost by over 20% on average (Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (opens in a new tab)). The emerging resolution is progressive disclosure: minimal base context, detailed context loaded on demand via skills, though the optimal balance is not yet established (AI Productivity Research Evidence).

Static documentation is meanwhile giving way to continuous generation. Google’s Code Wiki rescans a repository after each change and regenerates structured docs, architecture, class, and sequence diagrams, plus a chat agent grounded in the always-current wiki (Google - Introducing Code Wiki (opens in a new tab)). Every wiki section and chat answer hyperlinks straight to the underlying files, so reading and navigating merge into one workflow, and Google aims it squarely at legacy and private repositories where the original author may be gone, via a local Gemini CLI extension that keeps internal code on-premises (Google - Introducing Code Wiki (opens in a new tab)). The dark-factory “Living Portfolio” pattern applies the same idea at portfolio scale: every solution is catalogued and documented on creation, and maintenance agents generate documentation and changelogs from code and change history so the portfolio keeps itself healthy (Dark Factory Patterns).

The mechanism behind size discipline is attention: in very large context windows the model cannot tell which earlier tokens matter, so designers have to actively manage what stays resident (BRAID: bounded reasoning for agents (YouTube) (opens in a new tab)). A useful mental model treats context as a fixed allocation array whose first slots hold the harness prompt and the always-resident AGENTS.md; an oversized instruction file permanently steals working room and pushes the agent toward a lower-quality zone, which is the concrete argument for keeping it around seventy lines (Ralph and AGENTS.md tuning (YouTube) (opens in a new tab)). The craft is to write just enough to trigger the model’s latent knowledge rather than spelling out every step; naming journalctl leads the model to reach for systemctl on its own, so over-specification is counterproductive (Ralph and AGENTS.md tuning (YouTube) (opens in a new tab)).

TNG’s migration work reached the same conclusion from failure: overloading the model with irrelevant detail degrades basic task adherence, so start with only the necessary context and add more only where results fail (TNG - AI-assisted PL/I to Java migration (opens in a new tab)). The signal for when to update the file is behavioral: repeated tool-call failures mean the instructions, not the agent, need tuning (Ralph and AGENTS.md tuning (YouTube) (opens in a new tab)).

The instruction file is drifting from prose toward code. The System Initiative team treats CLAUDE.md as the executable contract and center of gravity of the system, holding terse constraints (TypeScript strict, named exports only, no fire-and-forget promises) rather than documentation (If humans architect the system, the AI writes the code (YouTube) (opens in a new tab)). One source argues against a single instruction file shared by every model. Because tone that helps one model hurts another (some models degrade under firm, uppercase language while others are tuned to respond to it), a single shared instruction file breaks and per-model files start to make sense (Ralph and AGENTS.md tuning (YouTube) (opens in a new tab)).

Several teams make the file self-improving: a standing line tells the agent to record any non-obvious trap and propose an update before continuing (If humans architect the system, the AI writes the code (YouTube) (opens in a new tab)), a design file carries an instruction to append any new design value it discovers so each session starts more refined (AI Labs - Three levels of AI UI design (YouTube) (opens in a new tab)), and a lightweight context.md read at session start and updated as state changes gives agents portable working memory with no code changes (Every - Agent-native Architectures (opens in a new tab)).

Prompts themselves can be structured artifacts: encoding a system prompt as a Mermaid flowchart gives the agent explicit branch pointers instead of a linear wall of text it must interpret, raising signal-to-noise enough that a small, cheap model can rival a top-tier one (BRAID: bounded reasoning for agents (YouTube) (opens in a new tab)).

Two second-order effects are worth naming. First, the agent is often the reason documentation finally gets written: conventions nobody wanted to author for colleagues get written down for the assistant, and humans navigate the codebase better as a result (INNOQ - Softwaremodernisierung mit GenAI (YouTube) (opens in a new tab)), while code readability now serves the agent as a first-class consumer, since clear intent makes code easier for the next agent to build on (Modern Software Engineering (YouTube) (opens in a new tab)). Second, this layer is an asset that is built, not installed; standing up the tool takes minutes, but the architectural memory, quality checks, and governance around it are built over quarters (Into the Wind: an architecture for AI transformation (opens in a new tab), 11 June 2026).

Sub-agents are becoming a portable format in their own right, authored to a fixed template (frontmatter, identity and memory, mission, domain rules, deliverables with examples, workflow, success metrics) and convertible across a dozen agentic tools from a single markdown definition (agency-agents (opens in a new tab)). A parallel effort pushes that portability into a Kubernetes-style envelope: AgentSpec defines YAML documents with apiVersion, kind, metadata, and spec across four kinds (Agent, Skill, MultiAgent, Workflow), where the spec carries intent (problem, outcomes, standard operating procedures, constraints, assumptions), context bindings to input data, actions with approval policies and error handling, capabilities, evaluation criteria with scenarios and edge cases, and a targets list that builds one definition out to several agent runtimes (AgentSpec (opens in a new tab), version 0.3.0 as of July 2026). The bet is that intent and evaluation belong in the same artifact as the prompt, so a definition ships with the criteria that judge it instead of acquiring them later.

Resources

Specification practice