Software Factory evidence and its limits

What public implementations establish

Public Factory-like implementations show that the approach is feasible. Agents can work in parallel, open pull requests, run checks, react to failures, deploy changes, and keep running after the merge.

They also show where the people go. Engineers spend more of their time specifying intent, designing environments, improving feedback, and deciding which output deserves a closer look.

What they don’t yield is one productivity number that transfers. Most reports come from model or platform vendors, from greenfield products, and from teams able to build custom infrastructure around their agents.

OpenAI’s internal product

OpenAI reports that it built an internal beta from an empty repository, without any code written by hand. Five months in, the repository held roughly one million lines and about 1,500 merged pull requests. The report cites an average of 3.5 pull requests per engineer per day and hundreds of internal users who used the product.

The most useful evidence is about operations. The team had to make the application, logs, metrics, and development environment legible to agents. Human QA capacity turned into a constraint. Engineers worked on scaffolding and feedback rather than contributing code themselves.

There is no controlled comparison for quality or business value in the report. OpenAI has direct access to its own models and a team picked for this experiment. The numbers describe what this system produced. They are no forecast for another organization.

Ona’s ten-day Factory

Over ten days, Ona reports 375 merged pull requests, more than 67,000 lines, 1,067 tests, and an 87 percent autonomous rate. Sixteen narrow automations handled planning, implementation, review, deployment, monitoring, and incident handling.

The architecture matters more than the code count. Ona didn’t run one general agent. It wired up bounded automations with clear triggers and handoffs.

The experiment was short, and it started from a blank slate. Repository activity does not establish useful product adoption, long-term changeability, or results in a regulated enterprise. Ona also sells the platform it used for the work.

Delivery research supplies the missing context

DORA and CircleCI study broader populations instead of a single Factory implementation. DORA links better results to the organizational system around the teams. CircleCI’s telemetry shows that, for the median team, a rise in development activity can come with weaker main-branch flow and lower stability.

Those findings are why implementation reports need delivery measures alongside output measures. A Factory can produce plenty of accepted changes and still shift the constraint into review, integration, recovery, or product decisions.

How hackers&wizards uses this evidence

hackers&wizards reads external reports as evidence for the category and its mechanisms. They are not evidence of a hackers&wizards client outcome.

Our public client testimonials come from Training, Pilot and Rollout engagements, and each one names the engagement behind it. None of them comes from an Assessment yet.

Each client engagement needs its own baseline, a named data source, and a business outcome. External benchmarks can inform the questions. They can’t answer them for the client.

What stronger evidence would look like

Stronger evidence for a Factory would tie a defined value stream to delivery and business measures over time. It would keep the baseline, show the intervention, record how review and failures behaved, and say what else changed over the same period.

It would also publish negative or neutral results. A credible body of evidence has to show where a Factory failed to improve flow, where the cost outweighed the benefit, and where human coordination stayed the binding constraint.