Skip to content

Knowledge base

Inference economics and infrastructure

Agentic workloads make inference cost usage-shaped: parallel agents and growing context drive token consumption, so the bill depends on tokens per completed workflow more than on the list price per token. Routing simple tasks to small models and reserving frontier models for the work that needs them keeps it in proportion.

Written by

Updated:

Tools 4

Our position. We choose the model per station, not per project: the strongest model plans, and a cheaper, faster one implements the reviewed plan.

Subsidized subscriptions hide the real price per token. The levers that last are which model a task gets, how much context it carries, and where and on what hardware the inference runs.

What does agentic inference really cost?

Developer-tool pricing has cycled through five phases: per-repository (GitHub 2008), per-seat enterprise, flat-fee AI augmentation (Copilot at $10/month), flat-rate with usage caps (Cursor’s 500 requests), and finally abstracted usage units that map to underlying compute (Tokenomics #1: The Pricing Evolution of AI Coding Agents (opens in a new tab)). The break point came when agentic usage made flat rates untenable: by June 2025 Cursor’s power users were chaining multi-step agents through huge context windows, forcing an overnight move to token-based billing, surprise bills over $1,000, and a public backlash (Tokenomics #1 (opens in a new tab)). The tools converged on different abstractions (Devin’s compute units, Lovable’s and Replit’s effort-based credits, Bolt’s token ceilings), but the lesson is uniform: pricing now has to reflect real compute costs, and hybrid base-plus-credits models give predictability without unlimited exposure (Tokenomics #1 (opens in a new tab)).

The same pressure hits teams from the consumption side. Agent swarms turn a developer who called a model a few times a day into 5-10 agents running parallel context windows over 1,000+ documents, making token consumption unpredictable and uncapped with no human in the loop (Tokenomics #1 (opens in a new tab)). Anthropic’s move of claude -p and the Agent SDK to metered pricing is read by practitioners as a signal that unmetered plans will not last, and a reason to drive per-token cost down across the stack before the rest follows; one practitioner’s Claude Code use, which fit inside a $200/month subscription, would cost about $10,000/month metered, while his open-model workers run at $25-35/day, or about $47 with interactive coding, which he still calls too much (Open Models Are Ready (opens in a new tab)).

Frontier per-token prices anchor the top of the market. Claude Opus 5.5 lists at $4 per million input tokens and $20 per million output, with cache reads at $0.20 per million, plus a 1.1x surcharge for US-only inference (Anthropic – Claude Opus (opens in a new tab), vendor, October 2026). Competing coding models undercut this, but total cost depends on how many tokens a model burns: reviewing Opus 4.6 against OpenAI’s Codex, the review quotes Codex at roughly half Opus’s per-token price, yet one benchmark run cost about $2,500 on Opus 4.6 versus $3,300 on Codex at full effort, dropping to $700 on Codex at medium effort while its score fell behind Gemini 3 Pro (Opus 4.6 vs Codex 5.3 (opens in a new tab), February 2026).

The prices users actually pay are heavily subsidized: one developer ran an estimated 1.3 billion tokens through a subscription plan without hitting 60% of quota, and the interview’s read is that providers “are subsidizing so much inference on these platforms right now” (Opus 4.6 vs Codex 5.3 (opens in a new tab)).

Inside teams that fund tokens centrally, the guidance inverts. Rather than rationing, the interview describes Shopify “effectively fund[ing] unlimited tokens for everybody” and steering engineers away from weaker models (“please don’t use anything less than Opus 4.6”), because a wrong model choice wastes more engineering time than tokens (AI infrastructure (opens in a new tab), April 2026). Code review is the expensive exception: it wants the largest frontier models taking turns rather than many cheap agents in parallel, a pattern that generates few tokens but runs slow and costly (AI infrastructure (opens in a new tab)). For engineering teams this argues for cost observability per workload and a routing strategy that reserves frontier models for the tasks that need them, which is the per-station split.

The continuous delivery community puts multiples on that routing decision. Output tokens cost two to five times as much as input tokens, and a frontier model costs ten to twenty times as much per token as a small one. Its routing ladder sends classification and extraction to a small model at 1x, summarization and formatting to a small or mid tier at 2 to 5x, code generation to a mid or frontier tier at 10 to 20x, and architecture review to a frontier model at 15 to 30x. Agentic workflows multiply the bill. Context grows at every orchestrator step, a change retried three times costs four times a clean success, and a failed correction loop burns about three times the tokens of a first-time pass. The guide therefore measures tokens per complete workflow rather than per call, and treats a run costing ten times the average as a design problem. Code quality drives cost as well: generic names, oversized functions and duplicated logic make the agent load more context and retry more often (MinimumCD.org and DojoConsortium.org - Tokenomics: Optimizing Token Usage in Agent Architecture (opens in a new tab), as of March 2026).

Budgeting for agentic workloads assumes today’s per-token prices hold, and practitioners planning against the curve do not assume that. One essay projects broad access within two to three years to systems producing tokens at 10x to 100x the current speed at a fraction of the cost, and expects that by the end of 2027 frontier-level intelligence will be cheap enough for most developers to work with full time, resting the projection on forthcoming accelerator and inference hardware rather than on model efficiency alone (The End of Programming (opens in a new tab), August 2026). That is a forecast, not a measurement, and the direction carries more weight than the multiple: a workload that is uneconomic at frontier rates today is a question of when it becomes economic, not a permanent constraint.

Where does inference run, and on what hardware?

Where inference runs is becoming as consequential as which model runs. Beyond the hyperscalers, an ecosystem of aggregators (OpenRouter), gateways (LiteLLM, Portkey), and sovereign regional clouds now sits between application and model. European providers such as Infercom host open-weight frontier models in EU datacenters for GDPR and AI Act compliance with no US CLOUD Act exposure (Infercom (opens in a new tab), vendor). Some run non-GPU silicon: SambaNova’s reconfigurable dataflow architecture claims up to 10x faster inference and 5x better energy efficiency than GPUs, and a heterogeneous blueprint splits the work, GPUs for prefill, SambaNova RDUs for decode, and CPUs for agentic tool calls (SambaNova (opens in a new tab); Infercom (opens in a new tab), vendor claims). Supply concentration is the systemic risk underneath all of it: a single Taiwanese foundry, TSMC, fabricates almost every leading AI chip (Stanford HAI – 2026 AI Index Report (opens in a new tab), April 2026), and a 2025 memory shortage drove DRAM prices up 19-24x (WTF Happened in 2025 (opens in a new tab)).

Not every inference workload runs a transformer. The interview describes Shopify serving a 300-million-parameter Liquid (state-space) model end-to-end in 30 milliseconds for search, and distilling large models into 7-8B task-specific versions that “very often beat” comparable open models like Qwen, while conceding such models will not match a frontier model head-to-head and are best treated as “a phenomenal target for distillation” (AI infrastructure (opens in a new tab), April 2026). For standard autoregressive serving, specialists such as Together, Fireworks, and CentML tune throughput to a given usage profile (AI infrastructure (opens in a new tab)). This matches the distillation trend an October 2025 interview on AI capability describes: as token usage explodes, distilling a large model into a small specialized one is increasingly the default path to cheap inference (AI capability interview (opens in a new tab), October 2025).

Resources

Delivery practice