Skip to content

Knowledge base

Coding benchmarks

Coding benchmarks measure whether an agent makes a small set of unit tests pass, mostly in Python and often on one function. They leave out the parts of the job with the most leverage: turning a fuzzy problem into a good specification, and keeping a codebase easy to change over many edits.

Written by

Updated:

Agentic Engineering 5

Our position. We hold that the harness matters more than the model (opens in a new tab). The model is a parameter (opens in a new tab), and the harness is the system the model runs in.

Benchmarks worth trusting control their task mix, keep headroom at the frontier and score cost, tool use and constraint adherence beside accuracy. The newer agent indexes also show one model scoring differently in different harnesses.

What do coding benchmarks actually measure?

Model cards quote coding benchmarks as if they measured software engineering, and they do not. Nilenso’s dissection of the popular ones is the corrective: they measure something narrower than their names suggest, and a model scoring 80% on SWE-bench does not one-shot 80% of real work. SWE-bench Verified is 500 problems, all Python, over 40% from the Django repository, with web applications entirely absent; the solutions are tiny (mean 11 lines, median 4, 77.6% touching a single function) and every issue predates 2024, so contamination is hard to rule out. Even the cleaner successors reduce to “make the unit tests pass,” which says nothing about whether the code is maintainable, secure, or correct; the UTBoost work shows many SWE-bench tasks pass their unit tests without actually resolving the underlying issue.

The high-leverage parts of the job, turning a fuzzy problem into a good specification and useful interfaces, are exactly what these benchmarks omit (What are popular AI coding benchmarks actually measuring? (opens in a new tab), 25 September 2025). The Python-centric bias is structural: AutoCodeBench was built specifically to counter it, generating 3,920 problems across 20 languages without manual annotation by verifying test cases through a multilingual sandbox, and even the strongest models struggle with its multilingual, high-difficulty tasks (AutoCodeBench (opens in a new tab), 12 August 2025).

Two failure modes recur once you leave the benchmark’s happy path. First, agents misuse real library APIs: a Tessl evaluation found standalone Claude Code and Cursor reached only about 60% accuracy on idiomatic use of open-source library APIs, with a marked drop for both legacy packages (released 2005 to 2011) and newly released ones past the training cutoff, plus a measurable bias against niche libraries; the fix in their study is versioned per-library documentation injected into the agent, and the vendor reports that its own documentation product lifts the agents’ abstraction-adherence score by about 35% relative to the same agents without it, but the transferable finding is that pre-training-led reasoning about APIs is unreliable at both ends of the age curve (Tessl – A Proposed Evaluation Framework for Coding Agents (opens in a new tab), 12 November 2025).

Second, low-resource enterprise languages expose the gap between a specialized model and a strong general one: on an ABAP benchmark that iterates with compiler and ABAP-Unit feedback for up to five rounds, SAP’s purpose-built ABAP-1 loses to general frontier models even at explaining code, and the practical lever is not another model but better tooling and feedback loops (static checkers, an ABAP MCP server) wrapped around a strong general model (Benchmarking LLMs for ABAP (opens in a new tab), 3 March 2026).

What separates a benchmark worth trusting from a vanity leaderboard is design discipline, and Snorkel’s meta-evaluation names the axes. On the science side: individual task quality (real-world complexity, verifiable solutions, validated by domain experts, with adversarial multi-reviewer protocols like GPQA’s); distributional control (a deliberate taxonomy that captures the rare-but-important failure modes, not just the common cases); difficulty and model headroom (unsaturated, so it actually separates the frontier, as ARC-AGI did); and eval methodology that goes beyond accuracy to cost, latency, reasoning-trace quality, tool use, and policy-constraint adherence, so a model that books the right flight but violates a fare rule still fails. On the art side: a thesis about where the field is going (Terminal Bench’s bet on the CLI), and researcher UX that makes the benchmark easy to run, extend, and contribute to (Snorkel AI – Meta-evaluations for building benchmarks (opens in a new tab), 4 June 2026).

A practitioner interview names a gap none of these axes close. Issue-to-patch benchmarks reward passing tests after one change, but they never test whether repeated changes keep a codebase easy to change months later. The interview proposes a longitudinal benchmark. An agent implements an unseen sequence of roughly twenty features, and the evaluator watches whether maintainability collapses before the sequence ends (Context engineering interview (opens in a new tab), 15 July 2026, practitioner and vendor). This is a practitioner proposal, not established evidence.

Artificial Analysis answers two of those axes, cost and harness, with a composite index for coding agents rather than for models. Version 1.5, current as of September 2026, averages three suites with equal weight, 113 long-horizon engineering tasks, 66 terminal tasks and 124 repository question-and-answer tasks, so the largest suite cannot outvote the smallest (Artificial Analysis - Coding Agent Index methodology (opens in a new tab)). Each task runs three times and scores pass or fail, a safety refusal that blocks the attempt counts as zero, and cost is the per-token API price of the tokens each task consumed, input, cache, reasoning and output tracked separately. Since v1.4 a judge agent reviews terminal trajectories and flags runs that edited the tests, read a reference solution or fetched the answer from outside (Artificial Analysis - Coding Agent Index methodology (opens in a new tab)). The index changes fast: v1.0 appeared in May 2026 and v1.5 in September, and scores from different versions are not comparable.

Because it plots score against spend, the index shows that the same model buys different results in different harnesses. In its comparison as read on 24 October 2026, Claude Code scored 62 at about $11.42 and 14.5 million tokens per task, while Codex scored 55 at about $7.04 and 3.2 million tokens (Artificial Analysis - Claude Code vs Codex (opens in a new tab)). Figures like these age within weeks. The lasting reading is that a higher effort setting buys points at a steeply rising price, so the question for a team is where on that curve its own tasks stop paying back.

Resources

Leaderboards