Skip to content

Knowledge base

Quality and Verification

Verification is the constraint of agentic delivery: when agents write most of the code, generation is cheap and checking is scarce. A team ships as fast as it can prove the output right, through tests, deterministic gates, acceptance evidence and evals, not as fast as it can produce code.

Written by

Updated:

Agentic Engineering 5

Our position. We do not let the model decide when its work is done. A guardrail (opens in a new tab) is a gate in code the agent cannot route around, such as a hook that runs the tests, never a line in a prompt asking it nicely. We set the limit on agent fan-out to what the team can check well (opens in a new tab), and raise it only when the checking improves.

The checking runs in layers. Weak changes are caught at authoring time, deterministic gates run from the language server through CI, acceptance evidence shows that a feature works, and outcome metrics show whether more output became more delivery. AI amplifies whatever quality practice a team already has, so these layers decide the result more than the model does.

Start here

Resources

Code Review & QA

Testing & Evals

Benchmarks

  • AutoCodeBenchmark (opens in a new tab) - Code generation benchmark with 3,920 problems for evaluating LLM coding ability.
  • CooperBench (opens in a new tab) - Stanford and SAP benchmark of 652 tasks measuring how well two coding agents cooperate, finding they succeed far less often than a single agent.
  • Harbor (opens in a new tab) - Framework for evaluating coding agents (Claude Code, OpenHands, Codex CLI) with benchmark authoring, parallel runs, and RL rollout generation.
  • How Tech Companies Measure AI Impact (opens in a new tab) - Pragmatic Engineer deep dive with a CTO interview on how GitHub, Google, Dropbox, Monzo, Atlassian, and 13 other companies measure AI tool impact on developers.
  • LLM SQL Benchmark (opens in a new tab) - Fast agentic text-to-SQL benchmark with a check-correct debug loop, scoring 25 questions on result correctness rather than the query text, runnable in-browser via DuckDB-WASM.
  • OpenRouter Rankings (opens in a new tab) - Weekly LLM rankings drawn from real OpenRouter usage, showing which models developers actually call.
  • SWE-Bench (opens in a new tab) - Leaderboard benchmarking AI agents on real GitHub issue-resolution tasks from open-source repositories.
  • SkillsBench (opens in a new tab) - Evaluation framework that scores AI agent skills across 84 expert-curated tasks, 7 models, and 3 agent harnesses.
  • Spark Arena (opens in a new tab) - Community leaderboard ranking LLM throughput in tokens/sec on NVIDIA DGX Spark across context lengths, concurrency, quantization (FP8, BF16, AWQ), and runtimes (vLLM, SGLang).
  • gbrain-evals (opens in a new tab) - Reproducible benchmarks for personal-knowledge and memory agent stacks (BrainBench plus LongMemEval) with sealed qrels, tolerance bands, and judge-version pinning to resist gaming.
  • live-bench (opens in a new tab) - LAN-only live benchmarking platform (FastAPI, click CLI, Vue 3 frontend) for streaming local-model evaluation results onto a presentation-grade dashboard.

Productivity Metrics

  • AI Engineer Coach (opens in a new tab) - Microsoft VS Code extension that turns local AI session logs from any harness into a dashboard with 45 anti-pattern rules and practice scores; fully local.
  • Bloom Filter AI (opens in a new tab) - Process intelligence that tracks how AI agents and humans execute work end-to-end across software teams.
  • DX Platform (opens in a new tab) - Developer experience measurement product built by DORA-style researchers to quantify productivity and the shift to AI-augmented engineering.
  • Engineering Social Graph (opens in a new tab) - Python tool that mines GitHub pull-request history to map developer collaboration networks, detecting teams via Louvain clustering, scoring expertise, and flagging bus-factor risk; MIT licensed.

Learning Platforms

  • Exercism (opens in a new tab) - Free practice-and-mentorship site spanning 79 programming languages with 7,948+ exercises, automated code analysis, and human mentors.
  • tests4geeks (opens in a new tab) - Coding-test service for hiring assessments across Java, PHP, C#, JavaScript, C++, Python, and SQL, with unlimited candidates, custom branding, and shareable test links.

In this section

  • Agentic continuous delivery

    The delivery rules for agent-written changes and what each pipeline stage checks, for teams letting agents commit to a production codebase.

  • Coding benchmarks

    Coding benchmarks measure whether an agent makes a small set of unit tests pass, mostly in Python and often on one function. They leave out the parts of the job with the most leverage: turning a fuzzy problem into a good specification, and keeping a codebase easy to change over many edits.

  • Deterministic quality gates

    How feedback loops from language servers through CI constrain agent behavior.

  • Evaluating non-deterministic systems

    How to test output that varies between runs, from unit tests to pass-rate baselines, and what a public eval leaderboard does and does not prove.

  • Quality and productivity metrics

    Under agentic coding, volume metrics such as lines of code or pull requests become harmful, because rewarding volume floods review with unchecked output. The measures that hold are first-pass acceptance, rework rate and the DORA delivery metrics, read for agent changes against the human baseline.

  • The surrounding quality system

    AI magnifies the engineering practices a team already has. Solid source control, observability and small batches turn into speed, while a review bottleneck or a brittle pipeline turns into longer delays and more breakage. The rules, reviews and hard checks around the agent steer what it produces.

  • The velocity trap

    Why generation can outpace verification and reduce real delivery throughput.