Skip to content

Knowledge base

Security of autonomous coding agents

A coding agent that can write files, run commands and reach the network can be turned against its user: a prompt injection that flips the agent's own approval settings becomes remote code execution. System-prompt instructions are no security boundary; concrete opt-in controls and a sandbox around the whole process are.

Written by

Updated:

Tools 4

Our position. We hold that neither a rule in a prompt nor a hook is a boundary: a hook checks the calls it was written for (opens in a new tab), and a shell has endless other ways to read a file. For what must never happen, we limit what the agent can reach at all, keep secrets out of the file system, and keep development keys apart from production keys.

The risk grows with autonomy. Each permission an agent gains, from writing its own config to running allowlisted commands, becomes a step an attacker can chain.

What attack surface does an autonomous coding agent open?

Autonomy widens the attack surface faster than most tooling accounts for. A security researcher shows that an agent able to write files without human approval can flip its own auto-approve setting (for example GitHub Copilot’s chat.tools.autoApprove), turning an indirect prompt injection into remote code execution; the same self-modifying-config flaw appeared in AWS Kiro and Amp and has since been fixed. The demonstration also covers cross-agent escape, where one agent writes a malicious MCP server into another agent’s config so the next launch yields RCE, and shows that allowlisted “safe” commands can be chained into data exfiltration, work that drew CVEs from Anthropic and an advisory from Amazon Q. The conclusion is that system-prompt instructions (“treat code as sensitive, never share secrets”) are robustness improvements, not security boundaries, and that sandboxes are insufficient because the secrets worth stealing live inside them (Agentic ProbLLMs: Exploiting Coding Agents (opens in a new tab), October 2025).

Which guardrails actually hold?

The account of building the pi coding agent reaches a blunter version of the same point from the builder’s side: because the combination of reading private data, executing code, and network access cannot be safely reconciled, most agents’ safety rails are “security theater”, so pi runs in full YOLO mode by default and tells users to sandbox the whole process if they need real isolation (pi (opens in a new tab), November 2025). Amp draws the operational consequence by moving permissions into a plugin, arguing it is now near-impossible to statically decide whether a tool call is destructive when frontier models write and chain throwaway scripts to get work done (Amp, Rebuilt (opens in a new tab), May 2026). Where guardrails do work, they are concrete and opt-in: Vercel’s agent-browser ships content-boundary markers that separate tool output from untrusted page content, a navigation domain allowlist, and an encrypted auth vault so the model logs in by name and never sees the password (vercel-labs - agent-browser (opens in a new tab)).

Resources

Articles

Posts

Talks and videos

Tools and repositories