AI writes your tests, but not the one for the right problem
A suite at 94% coverage, every test green, for a feature no user completed. The test that checks the business need depends on context only humans can supply.
Written by
-
Benedikt Stemmildt Founder & Co-CEO
Published: 13 March 2026 Updated: 2 min read
Every team has full test coverage. Almost nobody tests whether they solved the right problem.
AI writes tests effortlessly: unit tests, integration tests, edge cases, performance benchmarks. Green across the board. But there’s one test AI consistently fails at.
The test AI can’t write
It is the test that proves your system does what the business actually needs. Call it acceptance testing, behavior testing, or the “does this solve the real problem?” test.
Why AI fails here
AI doesn’t know what the business needs. It knows what you asked for. These are different things.
Last month in a training, a team showed me their AI-generated test suite. 94% coverage. Every test green. The feature had been live for two weeks, and zero users completed the flow. The tests confirmed the code worked. Nobody had tested whether it solved the problem.
The gap in a checkout flow
You ask: “Write tests for the checkout flow.” The AI delivers tests that checkout works technically.
What is missing are tests that checkout works for customers:
- Does the happy path match user expectations?
- Are error messages helpful, not just present?
- Does the flow make sense to someone who didn’t build it?
- Does the result align with the business outcome the feature was designed for?
What AI tests well, and what humans must own
AI tests these well:
- Code correctness
- Edge cases
- Error handling
- Performance benchmarks
Humans must own these:
- Business value alignment
- User experience coherence
- Requirement completeness
- Assumption validity
- Whether the feature solves the right problem
This is a context engineering problem
AI produces tests based on the context you give it. If you only give it code, it tests code. If you give it business requirements, domain constraints, user journeys and acceptance criteria, it gets closer.
The gap isn’t intelligence. It’s context.
Coverage metrics lie. You can have 100% coverage and still ship the wrong feature. The test that matters most, “Did we solve the right problem?”, requires context that lives in conversations, decisions and domain knowledge, not in the codebase.
Only humans can engineer that context. Only then can AI help write the test that matters.
Are you testing for coverage, or testing for value?
Further reading
Written by
-
Benedikt Stemmildt · Founder & Co-CEO
More on Specification and review
- Agent-written code breaks what code review was for Code review was never mainly about catching bugs. It kept the team's mental model aligned, and agent-written code breaks that loop before anyone has built the replacement.
- Cheap code makes engineering discipline worth more Generating code got cheap, and anyone can do it. Engineering systems that hold up in production is the skill that gained value.
- AI makes code easy, and only a spec keeps it simple Conversational coding piles every instruction onto the last until the code tangles. Rich Hickey's distinction between easy and simple explains why, and a research, plan, implement process is the fix.
Test the constraint in your system
The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.
Training on this topic: Agentic Engineering for Developers and QAWant to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check