Skip to content

Blog

A study says context files hurt agents. It tested the wrong thing

ETH Zurich found that auto-generated context files make agents worse. That shows auto-generation is useless, not context files. A useful context file records where the model fails your project.

Written by

Context engineering

Published: 24 February 2026 Updated: 2 min read

A new study says context files make coding agents worse. It tested the wrong thing.

ETH Zurich evaluated auto-generated context files across 138 tasks. Performance dropped in 5 of 8 scenarios, and inference costs went up by more than 20%. The researchers concluded that repository overviews in context files are not helpful.

What they actually showed is that auto-generated context files are useless. That is not news.

The flaw in the setup

The researchers generated the context files with the model. The model wrote down what it already knows, and then they fed that back to the model.

That is like writing yourself a reminder of your own name. Of course it does not help, and of course it wastes tokens. The information was already there.

What belongs in a context file

Context engineering is a reverse engineering process. You work with the model. You let it write code. You correct it, again and again, and you find the specific places where the model deviates from what you want.

Those deviations are your context file:

  • Behaviors the model gets wrong consistently.
  • Conventions that differ from the model’s training data.
  • Architectural decisions the model cannot infer from your code.
  • Anti-patterns specific to your codebase.
  • Defaults you need to override.

Only those belong in your context file. Leave everything else out.

Every model needs its own corrections

Every model has different defaults, different training data and different blind spots.

Your carefully crafted CLAUDE.md might need a complete rewrite when you switch models, or when a major update drops. Context engineering is not a one-time setup. It is continuous calibration against a moving target.

Why these results will not hold

The models in the study are already outdated. Current agents handle context files very differently, and running the experiment today would produce different results.

A study about AI that does not account for how fast the models change has a shelf life measured in weeks, not years.

The question the study should have asked

The study asked whether context files add information. The better question: do context files correct behavior the model gets wrong by default?

That is context engineering. Not templates and not auto-generation, but precise, targeted corrections discovered through iterative work with the actual model.

Write context files that describe where the model fails your project, not context files that describe your project.

Are you filling context files with what the model already knows, or with what it does not?

Further reading

  1. One AGENTS.md for four AI tools is harness debt A single context file behaves differently in every harness and every model. The four kinds of harness debt we keep finding in teams, and what a Dark Factory does instead. 2 min readShort post
  2. Context management matters more than picking the model Letting developers pick a model solves the wrong problem. The real constraint for coding agents is context, and AMP answers it with sub-agents, two working modes and fewer, sharper tools. 1 min readShort post
  3. What context engineering looks like in practice The gap between mediocre and strong agent output comes from how you instruct the agent, not from the model. Concrete practices across three layers: prompting, context and verification. 1 min readShort post

All 16 posts on Context engineering

ClientsCase study: A large DIY retailer in Germany

Test the constraint in your system

The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.

Training on this topic: Agentic Engineering for Developers and QA
See the assessment

Want to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check