Skip to content

Blog

AI writes your tests, but not the one for the right problem

A suite at 94% coverage, every test green, for a feature no user completed. The test that checks the business need depends on context only humans can supply.

Written by

Specification and review

Published: 13 March 2026 Updated: 2 min read

Every team has full test coverage. Almost nobody tests whether they solved the right problem.

AI writes tests effortlessly: unit tests, integration tests, edge cases, performance benchmarks. Green across the board. But there’s one test AI consistently fails at.

The test AI can’t write

It is the test that proves your system does what the business actually needs. Call it acceptance testing, behavior testing, or the “does this solve the real problem?” test.

Why AI fails here

AI doesn’t know what the business needs. It knows what you asked for. These are different things.

Last month in a training, a team showed me their AI-generated test suite. 94% coverage. Every test green. The feature had been live for two weeks, and zero users completed the flow. The tests confirmed the code worked. Nobody had tested whether it solved the problem.

The gap in a checkout flow

You ask: “Write tests for the checkout flow.” The AI delivers tests that checkout works technically.

What is missing are tests that checkout works for customers:

  • Does the happy path match user expectations?
  • Are error messages helpful, not just present?
  • Does the flow make sense to someone who didn’t build it?
  • Does the result align with the business outcome the feature was designed for?

What AI tests well, and what humans must own

AI tests these well:

  • Code correctness
  • Edge cases
  • Error handling
  • Performance benchmarks

Humans must own these:

  • Business value alignment
  • User experience coherence
  • Requirement completeness
  • Assumption validity
  • Whether the feature solves the right problem

This is a context engineering problem

AI produces tests based on the context you give it. If you only give it code, it tests code. If you give it business requirements, domain constraints, user journeys and acceptance criteria, it gets closer.

The gap isn’t intelligence. It’s context.

Coverage metrics lie. You can have 100% coverage and still ship the wrong feature. The test that matters most, “Did we solve the right problem?”, requires context that lives in conversations, decisions and domain knowledge, not in the codebase.

Only humans can engineer that context. Only then can AI help write the test that matters.

Are you testing for coverage, or testing for value?

Further reading

  1. Agent-written code breaks what code review was for Code review was never mainly about catching bugs. It kept the team's mental model aligned, and agent-written code breaks that loop before anyone has built the replacement. 2 min readShort post
  2. Cheap code makes engineering discipline worth more Generating code got cheap, and anyone can do it. Engineering systems that hold up in production is the skill that gained value. 1 min readShort post
  3. AI makes code easy, and only a spec keeps it simple Conversational coding piles every instruction onto the last until the code tangles. Rich Hickey's distinction between easy and simple explains why, and a research, plan, implement process is the fix. 2 min readShort post

All 23 posts on Specification and review

ClientsClient work

Test the constraint in your system

The Factory Readiness Assessment connects this research to your value stream, your data, and the next decision.

Training on this topic: Agentic Engineering for Developers and QA
See the assessment

Want to check one team first? The self-check: twelve yes or no questions, three minutes, instant result, no account. Take the self-check