Skip to content

Knowledge base

Choosing models

Choose a model by testing it on your own stack and tasks, not by its leaderboard score: coding benchmarks measure narrow, Python-heavy, well-specified issues, and capability varies sharply by language and task. Routing each role to a fitting model and investing in context and validation give more leverage than chasing the top model.

Written by

Updated:

Tools 4

Our position. We choose the model per station, not per project. Planning runs on the strongest model we have, and implementation runs on a cheaper, faster one that executes a plan already reviewed.

What do coding benchmarks actually measure?

Published coding benchmarks measure something narrower than their names suggest. SWE-bench Verified, the number most labs report, has 500 problems, all in Python, over 40% from the Django repository, with median solutions of 4 lines of code, and all issues date from 2023 or earlier, so the data was almost certainly in training sets (Nilenso - What are popular AI coding benchmarks actually measuring? (opens in a new tab)). A high score means the agent resolves well-defined issues with pointed requirements so that the unit tests pass; it says nothing about whether the code is maintainable, secure, or well-crafted, and nothing about the high-leverage work of turning a fuzzy business problem into a specification. This is why a model scoring 80% on SWE-bench does not one-shot 80% of real tasks (Nilenso - What are popular AI coding benchmarks actually measuring? (opens in a new tab)).

The narrowness matters more, not less, as scores saturate: SWE-bench Verified performance rose from roughly 60% to near 100% in a single year, and the US-China model gap has effectively closed, with the top US model leading by just 2.7% as of March 2026 (Stanford HAI - 2026 AI Index Report (opens in a new tab)).

The frontier is jagged, not uniform. A Gemini model earned a gold medal at the International Mathematical Olympiad while the best models read an analog clock correctly just 50.1% of the time (Stanford HAI – 2026 AI Index Report (opens in a new tab), April 2026), so a single “intelligence” score hides sharp per-task variance, and the interview cautions that benchmark-driven claims oversell real capability, with agents still producing “slop” on genuinely novel code (AI capability interview (opens in a new tab), October 2025).

Capability also varies sharply by programming language within the same model. A 17-model comparison across four typed languages found Sonnet 3.5 succeeding at 81% in Python and TypeScript but only 71% in Swift and 62% in Rust (LLM-Powered Programming: A Language Matrix Revealed (opens in a new tab)). Practitioner comparisons bear this out at the model level too: Codex is “phenomenal at Rust” but weak at Swift, while Opus, which “relies more on its training data,” handles fast-moving frameworks (Convex, Svelte, Tailwind v4) better and wins on greenfield work, whereas Codex wins in large existing codebases (Opus 4.6 vs Codex 5.3 (opens in a new tab), February 2026). Teams working outside the Python/TypeScript mainstream should evaluate candidate models on their own stack rather than trusting aggregate leaderboards; the benchmark ecosystem itself remains dominated by Python, with even improved suites like SWE-bench Pro covering only Python, Go, JS, and TS (Nilenso - What are popular AI coding benchmarks actually measuring? (opens in a new tab)).

How do you route work across models?

Per-role routing is starting to ship as configuration rather than as a hand-built pipeline. The pstack plugin’s setup command detects which models the user has access to and writes an always-applied rule mapping each role (code, judgment, and the multi-model review panels) to a model; every skill in the library reads that rule and falls back to defaults when it is absent, and the shipped default splits work by model strength, sending precisely specified code to one model, fast mechanical edits to a second, and prose and judgment to a third (cursor - pstack (opens in a new tab), as of August 2026).

The cheapest seat in such a routing table is now a model that generates no text. Jev, launched on 15 September 2026 as the first of what its vendor calls System One models, takes a state and a set of typed questions. It returns a choice from a list, a score on a scale, or the probability that a statement is true. It accepts text only, up to 64,000 tokens per request, and English is its primary training language (TypeSafe - Models (opens in a new tab), as of October 2026). A September 2026 product review measured the price on the reviewer’s own account: scoring a 2,000-word draft against four rubric questions took 2,994 input tokens and cost $0.000126, at a list price of $0.042 per million input tokens with output tokens free (Product with Attitude - What Is Jev? I Measured the Cost of a Judgment (opens in a new tab), September 2026).

The headline multiples are vendor self-claims with stated limits. The launch post gives 193.6 times faster and 444.6 times cheaper, measured on four workflow evaluations the vendor’s own team wrote. Reference answers were the average of two frontier LLMs, and the LLMs ran through a structured-output wrapper that the post says makes them slower and more expensive. The post adds that the price may be subsidized, and it carries no accuracy or calibration numbers of its own, pointing to a separate results site (TypeSafe - Introducing System One Models & Jev (opens in a new tab)). Vercel reports nearly 13% of paid teams on its AI Gateway using the model within 24 hours of launch, about twice the share of the GPT-5.6 family, and says the next test is whether that adoption lasts (Vercel - Jev is the fastest-adopted model in AI Gateway history (opens in a new tab), September 2026).

The vendor’s list of known failure modes is the better guide to what to route there. The model reads a question literally, cannot count, calculate, or compare dates reliably, loses accuracy as unrelated material fills the state, and in some cases leans toward the first option in a list. The stated remedies keep arithmetic and filtering in code, split a compound judgment into separate literal questions, and reorder the options to check that the answer stays the same (TypeSafe - Jev 1.13 jaggedness (opens in a new tab), reviewed 2 October 2026). Choice and score answers come with a confidence derived from the spread of their probabilities, while a true-or-false answer has only its probability. The vendor recommends acting automatically above a threshold that rises with the cost of a wrong action and routing low-confidence cases to a person or another system (TypeSafe - Confidence (opens in a new tab)). Calibration is measured across groups of predictions and does not guarantee that a single answer is correct (TypeSafe - System One (opens in a new tab)). How such a model works as a harness tool is covered in harness foundations.

Because no single model wins and behavioral differences persist, model-agnostic infrastructure is becoming the hedge. In a February 2026 comparison of 14 agentic engineering platforms, 10 supported multiple AI providers, to the point that model flexibility is table stakes and no longer differentiates; the report’s central premise is that the market is fragmenting and teams need platforms that work across providers (Model-Agnostic Agentic Engineering Platforms Compared (opens in a new tab)). This matches the evidence-side view: since frontier model quality converges and the infrastructure around the model determines outcomes, teams get more leverage from investing in context, tooling, and validation than from chasing the current leaderboard winner (AI Productivity Research Evidence).

Two arrangements pair a strong and a cheap model inside one task. The first is the per-station split: the strong model plans, writes a plan the executor can follow with no other context, and a cheaper model implements it step by step. The work returns to the strong model when the plan itself has to change (Augmented Coding Patterns - Smart Plan, Cheap Execution (opens in a new tab)). The second inverts the roles and qualifies the split: the cheap model runs the whole task and consults the strong one only when it is stuck. Anthropic shipped this as a beta API tool in April 2026 and reports, as a vendor self-claim, that Sonnet with an Opus advisor gained 2.7 points on SWE-bench Multilingual over Sonnet alone while cost per task fell 11.9%. Haiku with the same advisor reached 41.2% on BrowseComp against 19.7% alone. That is below Sonnet’s score, at 85% less cost per task than Sonnet. The advisor calls no tools and typically returns a plan of 400 to 700 tokens, which is why its share of the bill stays small (Anthropic - The advisor strategy (opens in a new tab), April 2026). For running the second arrangement by hand, a practitioner catalog names three escalation triggers: two failed attempts at the same problem, the agent reverting its own changes, and the agent declaring uncertainty (Augmented Coding Patterns - Advisor Strategy (opens in a new tab)).

Resources

Model classes

Model pairing