Just-in-time retrieval

Our position. We do not wait for bigger windows: for autonomous agents, long-context models are not the answer (opens in a new tab). The codebase is the state, the file system is the memory, and a sub-agent with a fresh window does the heavy reading and returns only the result.

The trade is speed against accuracy. Runtime search costs roundtrips and latency, so hybrids load some context up front and let the agent explore the rest. Where structure matters, as in counting queries or call chains across a large codebase, graph retrieval replaces similarity search.

Should an agent search on demand or index upfront?

The retrieval debate for coding agents runs between embedding-based pre-inference retrieval and agentic search. Cognition’s analysis of both: embedding search (RAG) is fast after indexing but inaccurate for queries that require tracing across a codebase, and can be counterproductive when the agent overweights irrelevant hits; agentic search with CLI tools is flexible and accurate but slow, burning dozens of sequential roundtrips, and forces the agent to attend to tens of thousands of irrelevant tokens, degrading answer quality. They observed agent trajectories spending over 60% of their first turn on context retrieval, and attacked the tradeoff with RL-trained retrieval sub-agents (SWE-grep) that issue up to 8 parallel tool calls per turn in at most 4 turns and return files with line ranges rather than free-form summaries, a verifiable output that keeps a fast model’s conclusions from misleading the main agent (Cognition - Introducing SWE-grep and SWE-grep-mini (opens in a new tab)).

Anthropic’s position has converged on just-in-time retrieval as the default for agents: instead of pre-processing all relevant data, the agent maintains lightweight identifiers (file paths, queries, links) and loads data at runtime through tools. Metadata does real work here; folder hierarchies, naming conventions, and timestamps signal purpose and relevance, and exploration enables progressive disclosure, assembling understanding layer by layer while keeping only the necessary subset in working memory. The stated tradeoff is speed: runtime exploration is slower than pre-computed retrieval, so hybrids retrieve some data up front and explore further at the agent’s discretion. Claude Code is the worked example, dropping CLAUDE.md into context naively while glob and grep retrieve files just in time, bypassing stale indexes and complex syntax trees (Anthropic - Effective Context Engineering for AI Agents (opens in a new tab)).

Manus pushes the externalization further and treats the file system as the ultimate context: unlimited, persistent, and directly operable by the agent, which reads and writes files as structured external memory. Their compression is designed to be restorable, drop a web page’s content but keep the URL, omit a document’s body but keep its path, so context shrinks without irreversible information loss, which matters because no one can predict which observation becomes critical ten steps later (Manus - Context Engineering for AI Agents: Lessons from Building Manus (opens in a new tab)).

Retrieval can return the wrong thing while looking like it worked. Analytical questions fail silently: a fixed top-N retriever asked how many tickets are open pulls ten, counts three done, and answers three when the real total is 137, because vector search is not built for aggregation, and the fix is a dedicated analytics query rather than more chunks (The Knowledge Gap - GraphRAG talk (opens in a new tab)). Similarity is not relevance either, “MySQL optimization” and “Postgres optimization” embed close together yet point at the wrong database, so a query-rewrite or split step earns its place (Wizard of Graph - Graph RAG talk (opens in a new tab)).

Chroma frames good retrieval as two stages, gather then glean: first maximize recall by pulling every possibly-relevant token even at the risk of noise, then maximize precision by cutting what does not belong, a pass cheap enough to brute-force with many small fast models in parallel (Chroma - Context Engineering for Engineers (opens in a new tab)). Some teams factor retrieval out as shared infrastructure rather than rebuilding a pipeline per agent, syncing sources continuously so embeddings do not silently go stale (Airweave (opens in a new tab), vendor).

At enterprise scale the case for structured retrieval sharpens. On a codebase past a hundred million lines, near-duplicate files across branches make embeddings collide, one file 99% similar and another 98.9% with neither clearly right, and brute-forcing a two-million-token window ran to about €20 and fifteen minutes per prompt while still producing un-debuggable code; the alternative parses the code into a graph and hands the agent only the relevant call chain, controller to endpoint to database (Rethinking Software with Dev Knowledge Graphs (opens in a new tab)). Graph retrieval trades latency for the multi-hop and counting queries embeddings cannot serve, climbing a ladder from loading a node’s neighbors, to letting the model choose which edges to follow, to a reasoning loop that re-traverses until the answer satisfies the question, roughly 200 milliseconds for the simple path against seconds for the loop (Wizard of Graph - Graph RAG talk (opens in a new tab)).

The recurring blocker is not the database but the ontology: taxonomies built by hand across departments go stale as teams split and merge, and a stale graph returns rubbish, which is why graph retrieval has landed more slowly than its advocates expected (Wizard of Graph - Graph RAG talk (opens in a new tab)). Indexing has its own ceiling, code-review platforms report mapping and indexing slowing sharply somewhere around 600 to 700 repositories (Qodo - Why More Context Makes Your Agent Dumber (opens in a new tab)).

One code-focused graph tool attacks that staleness blocker structurally rather than by curation. It builds a git-tracked folder of markdown nodes describing subsystems and APIs, joined by typed links (depends_on, part_of, uses) over a per-symbol graph extracted with tree-sitter, and deliberately omits embeddings and similarity search so the agent reads the linked files directly. The structural pass needs no model and no API key, LLM summarization is a separate optional pass cached by content hash, and a roughly 3-millisecond structural refresh re-touches only changed files, which makes the graph regenerable like a dependency folder rather than a hand-maintained taxonomy. Its own controlled trial (162 runs, one model held constant, varying only the context source) reports 42% fewer input tokens, 46% fewer tool calls, 60% lower latency and 32% lower cost against a cold start at unchanged 93% correctness, a tool-author benchmark rather than an independent one (Graft (opens in a new tab), MIT, as of July 2026).

A second line of attack targets the authoring cost rather than the refresh cost, and starts from structured data instead of code: read the table metadata already sitting in a lakehouse catalog, have a model draft an OWL ontology from it, map the classes back onto the tables with R2RML, and materialize the triples, so the ontology is generated from the warehouse it describes and re-synced from it instead of curated beside it. What that buys over the SQL schema is semantics the schema cannot carry, OWL 2 RL reasoning and SWRL rules to infer relationships that were never stored and SHACL shapes to validate the data against them, with the resulting graph exposed to agents over MCP. Maturity is early and stated as such: the implementation ships under a non-OSI vendor license, is labelled exploration-only with no support commitment, and states its capabilities without benchmarks (OntoBricks (opens in a new tab), v0.6.2, as of July 2026).

A context-engine vendor’s field account sharpens several of these points into design rules. Running the same real prompt (“make a Zendesk integration first-class”) once with naive per-tool MCP access and once through its engine, the naive PR compiled and passed every check yet, in the senior reviewer’s words, “would have broken our entire system if we had shipped it”, while the engine’s PR drew only a nitpick before merge, a concrete case of looks-right-but-wrong that mechanical gates miss. The diagnosis borrows a radiology term, satisfaction of search: a radiologist who spots one lung problem stops looking and misses the second, and naive retrieval stops at the first plausible match rather than the root cause, so retrieval has to be exhaustive rather than top-N.

Caching a correct answer for latency is an anti-pattern, because the moment it is written it can already be stale, and a cached answer replayed a day later effectively lies once the system underneath has changed. Truthiness is a live problem the engine only partly solves: when main-branch source conflicts with a Slack thread in which the CTO says it was built wrong, it weights the sources through a social graph of who wrote what and who they work with, which also carries the OAuth model through MCP for per-user governance, a concern that turns on past roughly twenty people. And bigger windows do not dissolve any of this: even a hypothetical hundred-million-token window only helps needle-in-a-haystack lookup, because a model cannot reason over that volume without entities and relationships to structure it (Unblocked – Stop Babysitting Your Agents (opens in a new tab), 26 May 2026).

The same vendor ships the engine as an MCP server and pitches it against the static rules file: instead of a hand-maintained instructions file, the agent queries a service that reads GitHub, GitLab, Slack, Confluence, Notion, Jira, Linear and Google Docs live, reconciles conflicting sources into one answer and enforces the asking user’s permissions on the server. Its page claims 60% lower token cost, 83% faster completion and nine of ten changes mergeable on the first pass, all vendor self-reports without a published method (Unblocked - Unblocked MCP (opens in a new tab), as of September 2026).

Qodo’s AI Engineer Europe talk compares the retrieval strategies above by what each costs to build and run, and the comparison reads as a sequencing guide. A context engine takes moderate indexing effort but slows down past several hundred repositories. Hierarchical summarization needs agents to re-summarize every file that changes, a high and recurring LLM bill. A knowledge graph asks the most of developers up front and pays off where logic spans many repositories. Iterative retrieval hands the agent an index to look things up in rather than a summary, costs more per query in recursive calls, and needs the least investment, which makes it the recommended starting point (Qodo - Why More Context Makes Your Agent Dumber (opens in a new tab), 8 June 2026). A self-correcting critic step that checks retrieved context against the original goal and retries adds latency but little setup.

Two open-source tools put the vector and the plain-file positions side by side. Zilliz’s Claude Context indexes a codebase into a vector database with AST-aware chunking, searches it with hybrid BM25 and dense-vector queries, re-indexes only changed files through Merkle trees, and reports about 40% fewer tokens at equal retrieval quality in its own evaluation. It needs a vector database and an embedding API key (zilliztech - Claude Context (opens in a new tab), as of July 2026). Vercel Labs’ opensrc takes the just-in-time route for dependencies instead: one command fetches and caches the source of any npm, PyPI or crates.io package or GitHub repository, so the agent greps the real library code rather than guessing its API from training data (vercel-labs - opensrc (opens in a new tab), as of September 2026).

Related

• Persistent agent memory: what an agent writes down so retrieval has something to find.
• Storage and retrieval architectures: vector, graph and file stores compared at organization scale.
• Designing efficient agent interfaces: how a tool returns only what the agent needs.
• Context and output interfaces: how a coding agent loads and returns context.
• Memory and Context Engineering: the wider discipline of managing what agents see.

Resources

Retrieval tools

• Claude Context (opens in a new tab) - MIT-licensed MCP server giving coding agents hybrid BM25 and vector search over a whole codebase, with incremental Merkle-tree reindexing.
• Unblocked MCP (opens in a new tab) - Commercial MCP server with a free tier that reconciles code, tickets, docs and chat into one permission-aware answer for coding agents.
• opensrc (opens in a new tab) - Apache-2.0 Vercel Labs experiment CLI that fetches and caches dependency source from npm, PyPI, crates.io and GitHub for agents to search.