Skip to content

Knowledge base

Evaluation and economics

Claude Code's capability stories run ahead of production reality, and evals on your own historical tasks are the corrective. Token cost explains much of the tool market around it: trackers, routing proxies and compressors exist because agentic coding broke request-based pricing.

Written by

Updated:

Tools 4

Our position. We measure token and context spend per operation before adding anything permanent (opens in a new tab). A token budget is an investment decision: it belongs with the people who decide what gets built, and it is judged by what comes out.

A toy version built in an hour is a starting point, and the production system it imitates still took a year (The Decoder – Google engineer says Claude Code built in one hour what her team spent a year on (opens in a new tab)). Reliability compounds per step, so a multi-stage workflow needs an engineered harness and measured evals rather than a better prompt (Harness engineering for reliable agent workflows (opens in a new tab)).

How far do capability stories hold up?

The ecosystem’s marketing runs ahead of its engineering, and the correctives are instructive. A Google principal engineer reported that Claude Code built from three paragraphs, in about an hour, a toy version of a distributed agent orchestrator the team had spent a year developing, then clarified it was a useful starting point rather than the production system, of which Google built several over the year with no clear winner; Google permits Claude Code only on open-source projects (The Decoder – Google engineer says Claude Code built in one hour what her team spent a year on (opens in a new tab), Jan 2026).

Thoughtworks’ reality check on Anthropic’s COBOL-modernization claims runs the same direction: mapping a legacy system is not just prompting Claude Code but a layered pipeline of chunking, local summarization, entity extraction, relationship inference, and synthesis, because models and agents cannot ingest tens of millions of lines regardless of context size; AI works with the static and dynamic analysis tools that already exist, not instead of them (Thoughtworks – Claude Code and COBOL modernization: What’s the reality? (opens in a new tab), March 2026).

The creator’s own trajectory is the cleanest capability story on record. Claude Code’s creator describes coding as an AI-safety petri dish (“it either compiles or doesn’t compile”), and dates the step changes to model releases: early Claude Code wrote 10-20% of the creator’s code, the jumps came with two later model generations, and after the second the creator had uninstalled the IDE, running five to ten Claudes in parallel, “I don’t prompt Claude anymore. I have loops that are running… My job is to write loops” (The origin story of Claude Code (opens in a new tab), 2 June 2026).

The team numbers attached: new-hire ramp-up fell from weeks to about two days, code per engineer grew roughly 3x, and roles are converging, every engineer scopes and talks to users daily, with the engineer-PM-designer distinction expected to dissolve; the founder advice is to shift budget from humans to tokens, staffing a four-engineer project with two engineers plus a large token budget. Even taste concedes: the no-classes codebase rule died when the model, by then writing all the code, kept writing classes, “okay well maybe the model’s right” (The origin story of Claude Code (opens in a new tab), 2 June 2026).

The non-coding capability story comes from Anthropic’s own growth team, whose head had to Google how to open a terminal when Claude Code launched and now ships a Figma ad-variation plugin built in about an hour (a 30-minute manual task cut to 30 seconds) and a Google-Ads skill that turns performance data into brand-voice ad variations; the team’s internal harness identifies growth opportunities, builds the experiment, tests it, and analyzes the results, and the working heuristic is “If you build a skill that sounds hyper-specific and a little boring when you describe it out loud, you probably built the right skill” (Anthropic’s growth team on Claude Code (opens in a new tab), April 2026). The same shape serves non-engineers outside Anthropic: a three-folder operating system (skills do the work, knowledge makes skills better, projects hold the output) with Obsidian as the human-facing viewer over Claude Code’s markdown and the whole knowledge base version-controlled in git like a codebase (A Claude Code operating system for non-coders (opens in a new tab), April 2026).

A vendor capability story stretches the skill format past coding into ML operations. Hugging Face’s hf-llm-trainer skill packages the decisions behind fine-tuning, which GPU fits a given model size, when to use LoRA versus full fine-tuning, and how to configure Hub authentication, so a plain instruction like “fine-tune Qwen3-0.6B on this dataset” drives Claude Code to validate the data, pick hardware, submit the job to Hugging Face’s cloud GPUs, monitor it, and push the finished model to the Hub, with a review step that shows the estimated cost before any paid job runs (Hugging Face – We Got Claude to Fine-Tune an Open Source LLM (opens in a new tab), Dec 2025, vendor). The same skills repository is portable across harnesses, installed in Claude Code through the plugin marketplace, discovered by Codex via AGENTS.md, and loaded by Gemini CLI as an extension, which frames a skill as a cross-agent artifact rather than a Claude Code-only one (Hugging Face – We Got Claude to Fine-Tune an Open Source LLM (opens in a new tab), Dec 2025, vendor).

Evaluation discipline is thin but forming. Back-testing agents on your own historical tasks beats public benchmarks, which one practitioner dismisses as marketing, and cheap surface metrics, how many tool calls, how many retries, how long a run takes, make a useful “agent smell” check (How Claude Code works (opens in a new tab), 26 December 2025). Thoughtworks’ evaluation frame adds drift to the usual criteria: judge agents on time and effort, cost, correctness (hallucination and faithfulness), and the variance you see when the same tool does the same thing twice; on that frame they rate Claude Code exceptional on time and cost, weaker on correctness and drift as of March 2026, and find open-source alternatives like OpenCode comparable when paired with state-of-the-art models (Thoughtworks – Claude Code and COBOL modernization: What’s the reality? (opens in a new tab), March 2026).

For the prompts, slash commands, skills and orchestration agents a team builds itself, the continuous delivery community offers one test: if you would notice a regression, it needs an eval. Model upgrades and changes to a system prompt or temperature fall under the same test. One-off queries, thin wrappers and display settings do not. The eval suite lives in the repository beside the plugin but outside what ships, with its own suites, graders, fixtures and reference solutions. An eval runner such as Promptfoo executes it at temperature 0. The guide treats the suite as the daily development loop more than as a release gate: run it, read the failing transcripts, fix the prompt, run it again (MinimumCD.org and DojoConsortium.org - Team AI Evals for Coding Tools (opens in a new tab), as of March 2026).

A context-tooling vendor’s CEO describes an on-ramp that starts from data a team already holds. An agent reads past pull requests, extracts the review comments and course corrections, packages them as a review skill, and wires that skill into CI. A daily job reads how developers answered the reviewer that day and amends the skill. To stop those amendments from breaking earlier behaviour, the same past pull requests yield eval scenarios, each a task to repeat plus judging criteria, which run like a test suite before every skill change. With the suite in place the team can also check whether a much cheaper model holds the same results (Modern Software Engineering - How to Actually Succeed with AI Development (And the Lesson Most Teams Miss) (opens in a new tab), September 2026, vendor).

An aggregated guide adds a cited correctness caution, an ACM 2025 study finding Claude Code can generate 1.75x more logic errors than human-written code (the claim is carried by the guide, not verified here) (Claude Code Ultimate Guide (opens in a new tab), July 2026). The fleet statistics quantify the gap from the practitioner side: reported agentic research completeness ran around 20% and context recall at best 70% until, by May 2026, newer frontier models pushed recall past 70%; the presentation concludes to shift work distribution toward planning and especially validation, and to move everything you can from agent to code, code that can itself be generated, with legacy-modernization harnesses built from language parsers (Tree-sitter, Joern, CodeQL) plus thousands of generated validation tests (Agentic Fleets für Software Development (opens in a new tab), May 2026).

Reliability math explains why harnesses beat prompts for multi-stage work. The talk’s march of nines: a ten-step workflow at 90% per-step reliability run ten times a day produces over six failures a day, and each added nine has to be engineered; an evaluation of 84 popular skills found they improve pass rates but stay well short of unattended business use (Harness engineering for reliable agent workflows (opens in a new tab), March 2026). The pattern that works wraps the agent in deterministic scaffolding: Stripe’s “minions” validate every Claude Code change against a subset of their three million tests and merge 1,300 pull requests a week, and the operating rule is to codify anything that must happen every single time in the harness, reserving skills for expanding capability in co-pilot use (Harness engineering for reliable agent workflows (opens in a new tab), March 2026).

The cheapest reliability lever remains verification: Claude Code’s creator recommends giving Claude a way to verify its own work, a feedback loop reported to double or triple output quality, with two concrete steps: give Claude a tool to see the output of its work, then tell Claude about that tool (The Decoder – Google engineer says Claude Code built in one hour what her team spent a year on (opens in a new tab), Jan 2026; Plan mode and CLAUDE.md interview (opens in a new tab), 1 April 2026).

Anthropic’s organizational guidance matches the harness thesis: the fastest rollouts front-loaded the infrastructure, sometimes one person wiring the tooling before anyone else touched it; an “agent manager” role, a hybrid PM-engineer function owning the Claude Code ecosystem, is emerging in several organizations; and configuration decays as models improve, warranting a meaningful review every three to six months, as when a hook that enforced p4 edit on file writes became redundant once native Perforce mode shipped (Anthropic – How Claude Code works in large codebases (opens in a new tab), May 2026).

Why do trackers, routers and compressors exist?

A large slice of the Claude Code ecosystem catalog (status lines, usage monitors, routing proxies, token optimizers) exists because agentic coding broke the pricing models of the previous era. Power users chaining multi-step agents with huge context windows made flat request-based subscriptions untenable, which forced Cursor’s abrupt June 2025 repricing; Claude Code itself sells tiered subscriptions with prompt allowances per 5-hour window, viable largely because Anthropic owns the underlying models (Tokenomics #1: The Pricing Evolution (opens in a new tab)). When usage is metered in opaque windows, burn-rate visibility becomes an operational need, hence the many monitors and menu-bar trackers in that catalog. Claude Code now generates part of this feedback itself: an undocumented /insights command produces a usage report that reads like manager feedback, profiling working patterns, flagging habits like abandoning conversations before concrete results, and emitting copy-pastable skill, agent, and hook suggestions; it is non-deterministic and appears to weight recent work heavily (Claude Code’s /insights (opens in a new tab), Feb 2026).

Waste is also structural, which is what the optimization tools target. Idle MCP tools have measured at about 12% of the context window before any work starts (Claude Code feature roundup (opens in a new tab), 15 August 2025), and an overgrown always-loaded CLAUDE.md taxes every session, the waste the reduce-and-delegate techniques of Context Engineering exist to counter (Claude Code). The proxies and routers extend the same economics to model choice, swapping backends or picking cheaper models per request; practitioners point ANTHROPIC_BASE_URL at backends like Moonshot’s Kimi K2 for bulk coding, claimed around 80% cheaper than the native model (AI Builder Club – Spec-driven Claude Code workflow (opens in a new tab), 24 July 2025).

Claudish is a concrete instance: it sits between Claude Code and the model API, translates Anthropic’s tool schema to and from OpenAI and Gemini formats, and lets one session mix providers per role, a native Anthropic model for planning and a cheaper or free OpenRouter model for bulk coding, so teams reuse existing subscriptions across 15+ providers and 580+ models instead of paying for several (Claudish (opens in a new tab)). The team-level takeaway: treat context and tokens as a budgeted resource with the same discipline as cloud spend, measure before optimizing, and be skeptical of headline reduction percentages, which are vendor-claimed and workload-dependent.

A reference multi-agent setup from the continuous delivery community turns that budget into session mechanics. Every context opens with the stable system rules and ends with the material specific to the current scenario, so the prompt cache keeps hitting. Before an item goes into an agent’s context, it has to pass one question: would leaving it out change what the agent does? Agents exchange JSON rather than prose. A session closes with a summary under 150 words, which replaces the conversation history for the next session. Three numbers get tracked: tokens per session, review block rate and tokens per retry cycle. Per-session cost rising while the block rate holds steady means the context is growing for no reason (MinimumCD.org and DojoConsortium.org - Coding & Review Setup (opens in a new tab), as of March 2026).

Two measurements ground that skepticism. CodeGraph’s cross-repo benchmark reports 58% fewer tool calls, 22% faster runs, and file reads near zero, but its author states plainly that token savings are scale-dependent, small and noisy on a modest codebase and material only at big-monorepo scale, and that its margins shrank on Opus 4.8 not from regression but because the newer model greps and reads efficiently on its own (CodeGraph (opens in a new tab), July 2026); a practitioner testing code-graph tools likewise reports real speed gains but nothing near an advertised 49x token reduction (Seven tools to stop burning tokens in Claude Code (opens in a new tab), May 2026).

Anthropic argues against pre-built indexes from the staleness side: by the time a developer queries an embedding index it reflects the codebase as it existed hours, days, or weeks earlier, returning functions the team has since renamed, which is why Claude Code bets on live agentic search (Anthropic – How Claude Code works in large codebases (opens in a new tab), May 2026). The subscription side of the same economics: one plugin roundup reports a $200 Max subscription consuming closer to $1,800 in raw token cost (Nine Claude Code plugins roundup (opens in a new tab), May 2026).

Resources

Evals