Green CI does not prove the live site works

CI went green. The pull request merged. Everyone moved on. And the live site was broken for two hours before anyone clicked it.

That gap is what the Post-Merge Verifier exists to close. It runs in the software factory built in public at software-factory.dev.

Triggered by a merge, not a schedule

It does not run on a schedule. The trigger is a single event: a pull request merged. The moment something lands on main, the Verifier wakes up.

Its first move is to do nothing. It reads the merged pull request and, if the title marks it as a chore, a docs change, or a refactor, it skips verification entirely. Those changes cannot affect the running app. For a feature or a fix, it continues.

Testing the real site

Then it waits 90 seconds for the deploy to reach production. It runs the end-to-end test suite against the real site. It runs smoke tests on top, hitting the landing page, the sign-in page, the health endpoint, and an authenticated flow. For a feature merge, it adds an interaction pass, clicking real buttons and opening real dropdowns and watching the browser console for errors. If anything fails, it files a priority-1 bug.

Two lines that keep failures real

Here is the detail worth copying, and it is two lines of script. Before the Verifier tests any route, it sends a HEAD request to that route first. If the route returns 404, it skips the check and does not count it as a failure.

That sounds defensive. It is what lets the automation run against a product still being built. A factory ships routes incrementally. A dashboard exists this week, did not last week. Without the guard, the Verifier would fail every run on pages that do not exist yet, and the team would start ignoring its reports. With the guard, a failure always means a real regression.

CI and production are two claims

The deeper point is the one in the automation’s own design. CI proves the code compiled and the tests passed. It does not prove the deployed site works. Those are two different claims, and a factory needs a part whose only job is to check the second one against reality.

You can read the whole automation in the Post-Merge Verifier definition on GitHub (opens in a new tab).

After your pipeline goes green, what actually confirms the live site still works, and how long would a break sit there unnoticed?

Further reading

• The surrounding quality system →
• Agentic Software Factory: Advanced Team Training →