Skip to content

Knowledge base

Contributor trust and dual use

AI makes plausible low-quality contributions cheap, so projects replace default trust in contributors with explicit vouching and proof that a human read the code. The same models let any engineer run a security review and let attackers find exploits faster than the industry patches.

Written by

Updated:

Software Factories 5

Our position. We treat agent security as a property you can inspect, not a vibe, and we do not consider it solved: it stays a standing human concern that gets sharper as agents get more capable (Agent security is not a vibe (opens in a new tab)).

The barrier to entry used to filter contributions, and coding agents removed it. Exploit search lost its barrier the same way, which turns patch speed into the deciding defense.

How do you decide whose code to accept?

Trust in who contributes code is changing alongside trust in what the code does. The Vouch model, first deployed in production in Ghostty, gates OSS contributions on explicit social trust: only vouched users may open PRs, maintainers vouch with an lgtm comment, and bad actors are denounced and blocked. The motivation is direct, AI tools let anyone “trivially create plausible-looking but extremely low-quality contributions with little to no true understanding,” so default trust based on the barrier to entry no longer works. The precise label is trust-based contribution gating, not reputation scoring: explicit, adversarial, and qualitative rather than an accumulated score (OSS Trust-Gating in the AI-Contribution Era). The constraint shifts from whether AI can write code to whether the organization can absorb AI-generated code safely.

A lighter gate works inside a single repository. One open-source linter project welcomes agent-written pull requests on the condition that a human reviewed the code, and makes the agent check that. Before submitting, the agent asks its operator one question about the code that only someone who read it can answer. Pull requests are accepted only for issues already assigned to the submitter, and the rest are closed automatically. Issues labelled as ready for unsupervised work are exempt from the human review, so a contributor can donate agent time without their own. An agent opens an issue only after its operator confirms, and the issue carries a review label and an opening disclaimer (habit-hooks - AGENTS.md (opens in a new tab), habit-hooks - Contributing (opens in a new tab), as of October 2026).

Does AI help attackers more than defenders?

Anthropic frames the larger picture as dual-use: better-aligned models let any engineer perform security reviews, hardening, and monitoring that previously required specialists, but the same capabilities scale attackers, so security has to be built in from the start rather than bolted on; the report expects agentic cyber defense systems that respond at machine speed (2026 Agentic Coding Trends Report). The offensive edge is already visible. Analyzing Anthropic’s Claude Mythos preview, which the company withheld from public release as too dangerous after it found thousands of zero-days, a 27-year-old OpenBSD bug, and a 16-year-old FFmpeg flaw that fuzzers had tested five million times without catching, a practitioner argues the results are mathematics rather than marketing.

What an LLM adds to exploit search is semantic understanding of code and the reasoning to act on it: hypothesis formation that prioritizes where implementation diverges from intent, depth of traversal past the three-derivation limit where humans and pattern-matching scanners both fail, semantic validation that eliminates the false positives that blunt static tools, and hard validation by writing a proof-of-concept exploit and running it in a sandbox. Agents scale horizontally, hundreds of instances across a project’s files turning a researcher’s weeks into hours (Claude Mythos vs Security: Marketing oder Realität? (opens in a new tab), 10 April 2026).

The primary source confirms the shape of that and is unusually candid about it. Anthropic published a system card for Claude Mythos Preview without releasing the model, a first, and says plainly that the decision was not forced by its Responsible Scaling Policy: the model demonstrated the ability to autonomously discover and exploit zero-day vulnerabilities in major operating systems and web browsers, the capability is inherently dual-use, so it went to a small set of partners for cyber defense instead of general availability. The benchmark picture explains the caution. Mythos Preview saturates Cybench, solving every tested challenge at 100% pass@1, to the point that Anthropic is questioning whether CTF-style benchmarks remain informative at all; on CyberGym, which tests reproduction of real vulnerabilities in open-source projects, it scores 0.83 against Opus 4.6’s 0.67 and Sonnet 4.6’s 0.65.

Anthropic’s own risk conclusion is that catastrophic risk remains low but that misalignment risk is higher than for previous models, that the model does not cross its automated-AI-R&D threshold but that this conclusion is held with less confidence than for any prior model, and, in the report’s own words, that it is alarming the world looks on track to develop superhuman systems without stronger industry-wide safety mechanisms (Anthropic, Claude Mythos Preview System Card, April 2026, and the accompanying assessment of its cybersecurity capabilities (opens in a new tab)). Read against the analysis’s patch-gap argument, the defensive-partner-first strategy is a race condition, not a solution.

The defensive story is less reassuring than the vendor framing. The claim that defenders get the capability first and harden critical infrastructure before open models catch up ignores the patch gap: agents produce validated exploits in days, faster than the industry patches in months, and a kernel or browser security update takes weeks to months of review, testing, and rollout, while embedded, legacy, and IoT systems take years or never update. When these capabilities reach open, Chinese, or ability-stripped models, attackers face an all-you-can-eat buffet of vulnerabilities in software that runs everywhere, so the practical question is less whether the announcement is marketing and more how many soon-to-be-unpatched systems run in your own network (Claude Mythos vs Security: Marketing oder Realität? (opens in a new tab), 10 April 2026).

Resources

Articles

Papers and reports

Posts

Tools and repositories