Build an auditor that improves your automations' prompts
An automation that rewrites other automations' instructions sounds risky. Built as a loop of real runs, explained verdicts and a reviewed diff, it is continuous improvement with one gate.
Written by
-
Benedikt Stemmildt Founder & Co-CEO
Published: 16 July 2026 Updated: 2 min read
In a real automation catalog there is one item that sounds like science fiction and is not: an automation whose only job is to read the other automations and make their instructions better. Teams hear “an agent that improves its own prompt” and picture something autonomous and a little dangerous. Built properly it is neither. It is a loop, and here is the loop.
The instruction set is the file teams never revisit
Start with the thing being improved. Every automation runs on an instruction set: a system prompt, a context file, a set of rules. Most teams write that once and never touch it again. It is the least maintained file on the line, and the one with the most effect on its output.
Step one: collect real runs
Take the automation’s actual output on real tasks, not toy ones. The PR reviewer’s last hundred reviews. The verifier’s last hundred checks. This is your evidence.
Step two: judge each run
Run every output past an evaluation that decides pass or fail against a real target. Did the test catch the bug? Did the review flag the actual risk? The trick that makes this work is in the next step.
Step three: demand the why
A pass-fail score alone is almost useless, the same way an exam grade with no feedback teaches a student nothing. Make the evaluation explain itself. Not “fail”, but “failed because it missed the null case in the error path”. The explanation is the signal. A bare score is not.
Step four: feed the explanations into a revision
Collect the failure explanations and hand them to a model along with the current instruction set. Ask for a revised set that would have caught those specific failures. This is the auditor doing its job.
Step five: review the diff like code
This is the step teams skip. The revised instruction set is a proposed change, and it goes through the same gate as any change: a human reads the diff, it is versioned, and it is tested against held-back runs before it goes live. The auditor proposes. It does not deploy.
One gate makes the loop safe
That last step is the whole safety story. An automation that rewrites instructions and ships them unreviewed is a risk. The same automation proposing a reviewed, tested diff is continuous improvement with the loop closed. The difference is one gate.
The payoff is better instructions with no new model. The instruction set was the variable all along.
Which automation on your line is running on a prompt nobody has improved since the day it was written?
Further reading
Written by
-
Benedikt Stemmildt · Founder & Co-CEO
More on Software Factory
- What is a Software Factory? A Software Factory coordinates agents, context, deterministic gates, and human decisions across the path from intent to production.
- Strong automated gates separate a dark factory from a bullshit factory Above a certain complexity, human code review lowers quality at agent speed. Quality inversion, the automation ratchet, and what a software dark factory needs instead of approvals.
- AI modernization works as a delivery problem, not a code translation Translating COBOL to Java one to one reproduces the old debt. Treated as delivery, with architecture decisions, encoded context and a migration factory, AI-driven modernization works.
Scaling needs named owners
Coding agents scale across teams when named people own the platform, the team practice, the guardrails, and the measurement. Rollout & scale is where that ownership is built and handed to internal owners, in your stack.
Training on this topic: Agentic Software Factory: Advanced Team TrainingWant to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check