An auto-merging PR reviewer is safe once it knows when to stop

A coworker asked me last week which automation in the factory scares people the most when they first read it. Easy answer. The PR Reviewer, because it merges code with no human approval at all.

The factory is built in public at software-factory.dev. The PR Reviewer is the automation that closes the loop on every pull request.

Two triggers, one action per run

It runs two ways. A pull-request event wakes it the moment something changes, and a cron line, */30 * * * *, sweeps every open pull request every thirty minutes in case an event was missed.

It works as a small state machine, and it takes exactly one action per run:

• If CI is failing, it reads the logs and pushes a fix.
• If CI is passing and the pull request has no review yet, it reviews it for correctness, security, conventions, and design.
• If CI is passing and the review is clean, it resolves the open review threads and merges.

It will not merge while a single review thread is still unresolved.

So far this is a competent reviewer that happens to have no human in the loop. The thing that makes it safe to run unwatched is one number.

Three attempts, then a human

Here is the detail worth copying. The PR Reviewer counts how many times it has already tried to fix CI on this same pull request. The marker is a ci-fix tag in the commit messages. After three attempts, it stops. It writes a comment saying CI has failed three times and needs human investigation, and it walks away.

That circuit breaker is what separates autonomy from a loop that burns. Without it, a pull request with a problem the automation cannot understand gets fix attempt four, five, six, forever, each push triggering the next run. With it, the automation has a known floor: three tries, then a human.

Autonomy needs a stopping point

The dangerous automation is the one with no number at which it admits it is stuck. Autonomy is safe exactly when it has a stopping point you can name out loud.

You can read the whole automation in the PR Reviewer definition on GitHub (opens in a new tab).

For the pull-request automations on your own floor, what is the number at which they hand the problem back?

Further reading

• Creator and verifier roles →
• Agentic Software Factory: Advanced Team Training →