Skip to content

Knowledge base

Production evidence

Reported software factories reach high autonomous throughput where specifications, tests and environments were strong before the agents arrived. Most numbers are self-reports from vendors or frontier labs, and cost, review capacity and long-term maintainability remain the binding limits.

Written by

Updated:

Software Factories 5

Our position. We read factory numbers by what reaches production and what breaks there, because those are the metrics that survive once automated gates replace manual review (AI measurement framework (opens in a new tab)). Productivity is not value: AI inflates everything easy to measure, such as code volume, pull request count and velocity, and is neutral on value (The productivity lie (opens in a new tab)).

Most reported deployments count pull requests and lines, so few of them answer that question. The sharpest failures appear where nobody reads the code the agents write.

What do factory deployments show?

  • Most public examples count pull requests, lines and autonomous rates. PFF is one of the few that reports a customer quality score.
  • The failures cluster where review stopped: agents imitate the patterns they find, so every sloppy change that merges becomes a template for the next ones.

Which deployments report numbers?

Concrete zero-human-code instances now size the pattern. OpenAI’s Codex team built and shipped an internal product with zero manually written lines, reaching roughly a million lines and 1,500 merged PRs in five months at 3.5 PRs per engineer per day, and, tellingly, throughput rose as the team grew from three to seven engineers rather than falling (Harness Engineering (opens in a new tab), OpenAI). Ona’s public Memo experiment shipped 375 merged PRs, 67,000 lines, and 1,067 tests across ten days at an 87 percent autonomous rate, with a median issue-to-close of 38 minutes and a median PR-to-merge of 4.9 minutes, run by 16 chained automations; its documented lesson is that spec quality is the control surface, most bugs were specification failures rather than capability failures, and a day of spec writing saves a week of bug fixing (Ona – We Built a Software Factory in 10 Days (opens in a new tab), 30 April 2026).

By September 2026 OpenAI runs the pattern across its whole engineering organization, and a site visit maps it as a nine-step loop. A human builder, an engineer or a product manager, defines the outcome, and OpenAI’s infrastructure leadership reports that this makes its engineers work more like product managers than systems engineers, because judgment, prioritization and taste decide the step. Codex then gathers context from the repositories, Slack, Notion, internal data sources such as Databricks and Datadog, and internal skills, some of which Codex maintains itself. OpenAI moved all its documentation into the source code so agents can read it, and new engineers are told to put their onboarding questions to Codex. The agent implements the change, builds and tests it, opens the pull request and fixes CI failures until the build is green, and a perf harness sends suspect changes to an A/B performance framework. Specialist review agents and a risk check decide the path to production, a deploy agent follows the change into production, and two agents feed operations back into development: Perf Factory for latency regressions and Sevbot for incidents (The Pragmatic Engineer - Inside OpenAI’s agentic software factory (opens in a new tab), September 2026). The internal Codex is wired into nearly every company system and is far ahead of the public product, so the setup belongs to a frontier lab with an unlimited token budget and does not transfer as it stands (The Pragmatic Engineer - Inside OpenAI’s agentic software factory (opens in a new tab), September 2026).

OpenAI's agentic software factory: a builder defines the outcome, Codex writes the code with context from internal systems, CI and specialist review agents gate it, a risk check routes it to agentic deploy or to an engineer's review, and Perf Factory and Sevbot watch production

Figure 1. Every change passes CI, specialist review agents and a risk check before deploy, and production monitors feed regressions and incidents back into Codex along the dashed paths. Figure from The Pragmatic Engineer - Inside OpenAI’s agentic software factory (opens in a new tab).

The same account measures where the constraint moved. Pull requests per engineer grow at an accelerating rate, and some build, test and deploy systems carry about ten times their former load. The infrastructure leadership puts that growth at two to three years for most companies and at about six months for OpenAI, and each new wave of model capability exposes the next bottleneck in version control, CI/CD or release. The engineering lead for ChatGPT names native mobile as an increasingly painful one: every app update waits hours to days for Apple’s and Google’s review, so code an agent writes in minutes still ships slowly, and web and mobile delivery speeds drift apart (The Pragmatic Engineer - Inside OpenAI’s agentic software factory (opens in a new tab), September 2026). IDE use inside OpenAI has fallen since January 2026, when Codex use surged, which is the bet the Codex team made when it chose not to fork VS Code for its desktop app (The Pragmatic Engineer - Inside OpenAI’s agentic software factory (opens in a new tab), September 2026).

The same shift holds at individual scale: one Ona build spent 30 minutes on a spec and 10 on execution, keeping the original spec as the clean source of intent while a separate iteration spec logged each problem and fix, on the principle that the bottleneck has moved from writing code to deciding what to build (Spec-Driven Design: Why Planning Is the New Coding (opens in a new tab), 25 March 2026). The same run marks the factory’s limits: it could satisfy requirements but drifted toward a generic default AI aesthetic and never produced taste on its own, which is why the human hours (8 to 10 a day at the start, building the harness of AGENTS.md, architecture.md, design.md, conventions.md, and a graded quality.md) dropped to 2 to 3 a day of metrics, review, and product direction once the machine was built (Ona – Building a Software Factory: Week 1 (opens in a new tab), 21 April 2026).

Every reports running four products with 15 people, roughly 99 percent of code written by agents, and argues the leverage is a step change at full adoption, a 10x gap between an org where 90 percent of engineers use AI and one where all of them do (Every – Compounding Engineering talk (opens in a new tab), 18 December 2025). A conference talk by a mid-size company reports its own numbers on the same shape, from a comparison of two teams rather than a controlled study: sports-data company PFF ran two of its strongest engineers in an agents-first workflow against a roughly ten-engineer team from January to March and measured 25 times the deploy frequency (five a day versus one every five days), 10x output on a blend of ticket count and code complexity, features shipped in under two months against a four-month estimate, and a customer quality score of 8.6 out of 10 versus roughly 7 to 7.5 before; the process finding is that “Scrum did not survive”, with no PM, no sprint planning, no standups (tickets auto-update from PR status), and no estimation, replaced by a spec-to-lightweight-design-document flow where the agent interviews the requester, auto-created non-blocking tickets, and a QA agent that checks acceptance criteria on every staging deploy (PFF agents-first case study talk (opens in a new tab), 15 May 2026).

Blitzy reports autonomously rewriting dnsmasq from C to Rust (86,000 lines, 592 tests passing, zero compilation errors) in three days and curl (215,000 lines, 7,312 tests, clean under Miri and AddressSanitizer) in five, placing the human gate at approving the technical specification rather than reviewing code and delivering about 80 percent of a project for humans to finish the last 20 (Blitzy – C to Rust: curl (opens in a new tab), vendor); on the same platform a designer with no coding background took a blog from Figma frames to a production deploy (Blitzy – How Blitzy Built This Blog (opens in a new tab)).

It also built a C compiler from an empty repository (229,983 lines of Rust, 2,271 tests, four backends, zero external dependencies, the Linux kernel booting on QEMU) after first fixing Anthropic’s own agent-built compiler, whose public release, though it compiled the Linux kernel, opened with a “hello world does not compile” issue and 14 miscompilations found by an external fuzzer, illustrating the gap between passing a test suite and production software (Blitzy – Hello World! The Blitzy C Compiler Has Arrived (opens in a new tab), vendor).

Blitzy frames the benchmark takeaway directly, a record 66.5 percent on the audited SWE-Bench Pro public set (against GPT 5.4 at 57.7 and Claude Code Opus 4.5 at 55.4), as evidence that advances at the harness layer now unlock larger gains than the model, because its hours spent building a whole-codebase knowledge graph let it get the boundary conditions a single-pass model misses right (Blitzy – Blitzy Scores a Record 66.5% on SWE-Bench Pro (opens in a new tab), vendor). Blitzy’s strategic claim behind those runs is that autonomy cannot be retrofitted onto an assistant: understanding and orchestration have to be fused from day zero, because “the graph that grounds a single agent in a single file is the same graph that lets a thousand agents maintain coherence across a million-file repository”; its enterprise evidence includes over a billion lines of code ingested since September 2025, a Fortune 100 reverse-engineering of 33 million lines of mainframe code in 3.5 days against a nine-month dedicated-team estimate, and a claimed 2.91 dollars of ARR generated per dollar burned since January 2025, with the corollary that once engineering stops being the bottleneck “the question stops being ‘can we build this?’ and starts being ‘what should we build?’” (Blitzy – Taking the Long and Less Traveled Road Is the Only Path to Autonomy (opens in a new tab), vendor). The economic case it presses is the same stranded-capital argument at project scale: a memory-safety rewrite like zlib to Rust is normally uneconomic (scarce Rust talent, a 30 to 50 percent velocity drop in the first months, only 15 percent of enterprise migrations finishing on time), and a factory that compresses those months is what makes the “SaaSpocalypse” of in-housing dissatisfied legacy ERP and SaaS plausible, against a baseline where Gartner reports 70 percent of ERP migrations fail their original goals (Blitzy – Harbinger of the SaaSpocalypse (opens in a new tab), vendor).

Augment Code frames the organizational answer as Cosmos, an org-level operating system coordinating humans, agents, code, tools, policy, and memory in one environment (vendor self-claim).

What does a factory cost to run?

The cost side is quoted in unit terms: StrongDM offers a rule of thumb that a factory with headroom spends at least 1,000 dollars of tokens per engineer per day (StrongDM - Software Factory (opens in a new tab)), and by February 2026 a practitioner account claimed the all-in cost of software development has fallen to about 10.42 dollars an hour, below minimum wage (Software Development Now Costs Less Than Minimum Wage (opens in a new tab), 27 February 2026). The cost base is also a dependency risk: with the frontier labs heading toward public-market scrutiny, pricing, packaging, quotas, and usage restrictions will keep changing, so the factory hedge is hybrid model routing, expensive frontier models for planning and hard reasoning, cheaper or local models for routine execution, review, classification, and mechanical work (What’s the Appeal of a Software Factory? (opens in a new tab)).

The same routing logic runs the other way at the review seam, where models are better at judging than generating, so a code-review or validation pass can drop a whole capability class below the generator and still hold; the factory is in part what keeps cheap, low-tier models on the rails, so the durable practice is to tag each unit of work with an intelligence tier and route it to the cheapest model that can do it, reserving frontier models to plan and to keep the fleet on the rails (Inside a Software Factory (opens in a new tab), 6 July 2026).

Where does lights-off autonomy fail?

One practitioner account puts a concrete limit on that autonomy: a July 2025 lights-off experiment ran until an issue required understanding a codebase no one had been reading, after which the site was down and the human had to return to the code. The explanation is structural: coding-agent training rewards passing tests without a comparable signal for architecture or maintainability, whose costs surface months later, so current benchmarks cannot establish that a model preserves a codebase over time (Harness engineering is not enough: Why software factories fail (opens in a new tab), July 2026). The proposed response is to keep human review and spend a short alignment phase on product intent, system architecture, types, call graphs, and vertical slices before implementation; the talk reports that roughly 30 minutes of planning can save hours of review, but this remains a practitioner claim rather than controlled evidence (Harness engineering is not enough: Why software factories fail (opens in a new tab), July 2026). The written version sizes the resulting trade explicitly: the reviewed workflow is offered as a 2-3x gain against the 10-100x the lights-off pitch promises and has not delivered, which is the honest number to hold a factory business case against (Why Software Factories Fail (opens in a new tab), July 2026).

A later interview with the same practitioner adds why skipped review compounds. Agents imitate the patterns they find in the codebase, so every sloppy change that merges becomes a template for the next ones. Both sides of the conversation read this against the view that technical debt is cheap to carry because agents will pay it down later, which would hold only if each change were written without reading existing code (AI Native Dev - Why We Stopped Trusting AI to Write the Plan (opens in a new tab), September 2026, practitioner and vendor). The missing benchmark got a first measurement in March 2026. SlopCodeBench makes an agent extend its own earlier solution across 196 checkpoints in 36 problems. In its May 2026 revision no agent solved any problem end to end, and the best passed 14.8 percent of checkpoints. Structural erosion rose in 77 percent of trajectories and verbosity in 75.5 percent. Agent code ended 2.3 times as verbose and 2.0 times as eroded as 473 open-source Python repositories, and explicit quality guidance lowered the starting level by up to a third without slowing the decline (SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (opens in a new tab), March 2026, revised May 2026).

What do self-merging and orchestrated builds measure?

A five-person company reports the self-merging end of the range. Agents write every line, CI gates and AI review bots run on every pull request, and a change that passes them merges itself, with the agent pushing its own fixes when a reviewer blocks. Merge is not release: a separate user-acceptance suite catches regressions daily and blocks every release until they are fixed, and its repository tells the agent that the tests are the source of truth and must not be changed. Over 30 days, 217 issues shipped and 81 were closed as duplicates or declined. The median time to triage was 4.6 elapsed hours and triage to shipped took 1.6, at about 3,000 dollars a month for five subscriptions plus the CI review spend. All figures are the speaker’s self-report from a conference talk (AI Native Dev - GitHub Next & Tessl on the Self-Merging Repo (opens in a new tab), September 2026, practitioner).

Factory published the full ledger of one orchestrated build, a Slack clone with channels, threads, reactions, file uploads, search and presence. The mission ran 16.5 hours across six milestones: orchestration took 2.3% of the time, implementation 60.5% and validation 37.2%. One orchestrator, 63 worker runs and 27 validator runs consumed 778.5M tokens, of which 744.9M were cache reads. The result was 38.8k lines of code, 52.5% of them tests, at 89.25% statement coverage. Validators surfaced 81 issues, and the orchestrator answered them with 21 fix features on top of 40 original ones, so milestones needed two to four validation rounds to pass. Median runs stayed at 51 assistant turns for implementation and 30 for validation. The workers shared state through files, including a validation contract, a feature list, a services file and AGENTS.md (Factory - How Missions Work (opens in a new tab), April 2026, vendor).

Resources

Factory evidence