Software Factory evidence and its limits

What public implementations establish

Public Factory-like implementations establish feasibility. Agents can work in parallel, open pull requests, run checks, respond to failures, deploy changes, and continue operating after merge.

They also show where people move. Engineers spend more time specifying intent, designing environments, improving feedback, and deciding which output deserves attention.

They do not establish one transferable productivity number. Most reports come from model or platform vendors, greenfield products, and teams that can build custom infrastructure around the agents.

OpenAI’s internal product

OpenAI reports building an internal beta from an empty repository with no manually written code. After five months, the repository had roughly one million lines and about 1,500 merged pull requests. The report gives an average of 3.5 pull requests per engineer per day and says hundreds of internal users used the product.

The most useful evidence is operational. The team had to make the application, logs, metrics, and development environment legible to agents. Human QA capacity became a constraint. Engineers worked on scaffolding and feedback instead of contributing code directly.

The report does not provide a controlled comparison for quality or business value. OpenAI has direct access to its models and a team selected for this experiment. The numbers show what this system produced, not what another organization should forecast.

Ona’s ten-day Factory

Ona reports 375 merged pull requests, more than 67,000 lines, 1,067 tests, and an 87 percent autonomous rate over ten days. Sixteen narrow automations covered planning, implementation, review, deployment, monitoring, and incident handling.

The architecture matters more than the code count. Ona did not run one general agent. It connected bounded automations with clear triggers and handoffs.

The experiment was short and greenfield. Repository activity does not establish useful product adoption, long-term changeability, or results in a regulated enterprise. Ona sells the platform used for the work.

Delivery research supplies the missing context

DORA and CircleCI examine broader populations rather than one Factory implementation. DORA associates better results with the surrounding organizational system. CircleCI’s telemetry shows that increased development activity can arrive with weaker main-branch flow and stability for the median team.

These sources explain why implementation reports need delivery measures beside output measures. A Factory can generate many accepted changes and still move the constraint into review, integration, recovery, or product decision-making.

How h&w uses this evidence

h&w treats external reports as evidence for the category and its mechanisms. They are not evidence of an h&w client outcome.

Our public client testimonials come from Training, Pilot, and Rollout engagements. Each one carries the matching product tag. The internal record keeps the narrower claim and limitations. No current testimonial belongs to Assessment. The internal Factory dashboard is an operational snapshot, not a client result or a live feed.

A client engagement needs its own baseline, named data source, and business outcome. External benchmarks can inform the questions. They cannot answer them for the client.

What stronger evidence would look like

Stronger Factory evidence would connect a defined value stream to delivery and business measures over time. It would preserve the baseline, show the intervention, record review and failure behavior, and state what else changed during the same period.

It would also publish negative or neutral results. A credible evidence base needs to show where a Factory failed to improve flow, where cost outweighed benefit, and where human coordination remained the binding constraint.