Fix the incident and the rule in the same commit

Most incident response stops at the fix. The factory built in public at software-factory.dev has an automation called the Incident Responder, and it goes one step further, in a way I want to walk through: when a bug reveals a gap in the rules, it patches the rules. You can read the Incident Responder automation definition on GitHub (opens in a new tab).

An hourly sweep of new errors

Start with the trigger. A cron line, “0 * * * *”, so it sweeps production once an hour. That hourly sweep is the only way in: no webhook, no manual run.

It reads the error tracker for unresolved errors first seen in the last 24 hours, with stack traces. Errors that already have a tracking issue are skipped, so it never triages the same thing twice.

For each new error it makes a call:

• Critical or high (a crash, data loss, an auth bypass) gets fixed now.
• Low (an edge case or something cosmetic) becomes an issue for later.

The full loop for a critical error

For a critical error it runs the full loop. It finds the root cause, opens a fix branch, writes the fix, adds a regression test that would have caught this exact error, runs the full check, and opens a pull request.

The rule update goes into the same commit

Here is the detail worth copying. Step six of the fix instructions is one sentence, and it is the whole point.

If the bug reveals a missing convention, an unhandled error path or a missing null-check pattern, the automation updates the conventions document in the same commit as the fix. The same commit as the fix, not a follow-up issue someone gets to later.

So the next time any automation reads the conventions doc before writing code, the rule that would have prevented this bug is already there. The fix closes the incident. The doc edit closes the class.

An incident is the cheapest moment to write the lesson down

This is the part I think teams underrate. A fix without a rule update waits to be broken again. The conventions doc is the factory’s memory, and an incident is the cheapest moment to write to it. You have just paid for the lesson in production, and the cause is right in front of you.

When something breaks in production, where does the lesson get written down, and does anything actually read it back before the next change ships?

Further reading

• The disciplined compounding loop →
• Agentic Software Factory: Advanced Team Training →