Verification in a Software Factory

Generation moves the constraint

Code generation can push more changes, and larger ones, into a delivery system. Review, integration, and product decisions don’t scale up with it automatically.

What you get is a queue. Pull requests sit longer, checks run more often, and people face more plausible output than they can judge. Before a Factory adds autonomy, it has to get better at verification.

Deterministic gates establish fixed facts

Builds, type checks, linters, tests, schema checks, security scans, and policy rules each answer a bounded question. Give them the same input and they should return the same verdict.

They are strongest when they guard a contract that existed before anyone wrote the implementation. A test written after the change can encode the same misunderstanding the code does. Where the risk justifies it, acceptance evidence should stay independent of the change.

Good independent evidence is an existing API contract, a recorded behavior comparison, a golden output, a property that must hold, or an end-to-end scenario maintained outside the implementation task.

Agent review has a different job

Agents can review intent, architecture, security, maintainability, and missing tests at a volume a single human cannot match. Their verdicts are still probabilistic. Several reviewers can repeat the same wrong assumption, or bury the correction loop in broad suggestions.

A review stage in production needs a shared severity vocabulary and a bounded scope. Critical findings can block the change, and warnings can send it back for rework. Questions should go to a person. Suggestions shouldn’t draw on the same budget as a broken contract.

Review loops also need a point where they stop. A loop that cannot converge after a few attempts has found uncertainty. That is no invitation to keep spending tokens indefinitely.

Green does not mean correct

A change can pass every automated check and still solve the wrong problem. It can preserve safety properties and still miss the behavior that makes it useful. It can pass unit and integration tests and slip in a product decision nobody intended.

Intent stays with people, and so does the judgment the gates cannot encode. The Factory should bring forward the work that deserves that attention, so that people don’t have to read every generated line.

Gate by reversibility and impact

Not every change deserves the same process.

A local refactoring with no public contract is cheap to undo. A database schema, a public API, a permission boundary, or a migration of production data gets expensive once other systems depend on it. Human attention belongs where a wrong decision is hard to reverse.

In practice that means operating modes. Low-risk work can go through the automated gates and merge. Higher-risk work can require an approved plan, a reviewed contract, or a person at deployment. The organization owns those boundaries.

Make the Factory legible

Agents need to see the application state they are supposed to evaluate. OpenAI describes making user interfaces, logs, metrics, and traces legible to Codex inside isolated worktrees. With that, agents could reproduce failures and check runtime behavior rather than stopping at the source code.

Each run should leave evidence behind: the original intent, the plan, the changes, the gate results, the review findings, the deployment state, and the cost. A failure that repeats can then turn into a durable rule or test.

How much a Factory can verify sets how fast it can safely go. More agents help only when the evidence system can keep up.