Software Development
AI-Generated Tests: Coverage Rose, Mutation Score Did Not
September 202612 min read
Because the agent read the implementation before writing them, so the expected values in the assertions are the values the code already produces. A test derived from the code cannot disagree with the code. If the implementation contains a bug, the test records that bug as the expected result and reports green.
No. Statement and branch coverage measure whether a line executed, not whether any assertion checked the result, and the assertion is not an input to either calculation. A test that runs every branch and only asserts that the return type is a number earns full coverage while verifying nothing at all.
The tool makes one small valid change to your source at a time, such as flipping a comparison operator or replacing a condition with true, and re-runs the suite against each modified copy. If a test fails, that mutant is killed and some assertion genuinely depends on the line. If every test passes, the mutant survived and the line is covered but unverified.
Usually yes, and that is why the tools all ship incremental modes and scoping options. Run it on the modules where a wrong answer costs money or trust, such as pricing, tax, approval thresholds and permissions, and scope it with the mutate array in StrykerJS, targetClasses in PIT or source_paths in mutmut. Nightly beats per-commit until you have measured the real wall-clock cost.
Withhold the implementation and give it the requirement instead, including every boundary value and the invariants. Then ask it to write out two plausible wrong implementations and confirm that at least one of its tests fails against each. Any test that passes against both the right and the wrong version is a surviving mutant found before the code was even written.
Key Takeaway
An AI agent asked to write tests will assert what the implementation currently does, so coverage rises while verification does not. Mutation testing exposes it: deliberately break a line and see whether any test fails. A surviving mutant marks code that is covered but unverified. Fix the cause by prompting from the specification, not the code.
The module was an approval router in an ERP, its coverage number had been embarrassing for a year, and I asked an agent to fix that. It read the file, wrote forty-one tests, and every single one passed on the first run. Two weeks later an amount of exactly fifty million routed to the wrong approver, at a boundary three of those new tests exercised by name.
The tests were not sloppy. They were circular, which is a different and much quieter failure: a test derived from the implementation cannot detect that the implementation is wrong, because it has already recorded the wrong answer as the expected one. This post is the mechanism behind that, the four shapes it takes in real code, the tool that catches it, what that tool honestly costs to run, and the one instruction in the prompt that stops the whole pattern.
Ask an agent to test a file and it will read the file. That single fact produces everything else. The expected values in the tests it hands back are the values the code produces, because the code is the only source of truth it was given, so the suite is green by construction. It would have been green against a different implementation too, as long as that implementation was the one it read.
// src/approval/level.ts
//
// Requirement, from the approval matrix in the spec:
// level 1 up to and including 10,000,000
// level 2 above 10,000,000 up to and including 50,000,000
// level 3 above 50,000,000
//
// The implementation below is WRONG at both boundaries: it uses > where the
// requirement says "up to and including", so an amount of exactly 50,000,000
// routes to level 2 instead of level 3. One character per branch.
export function approverLevel(amount: number): number {
if (amount > 50_000_000) return 3;
if (amount > 10_000_000) return 2;
return 1;
}Notice what such a suite can and cannot do. It can pin current behaviour, so a later refactor that changes the routing will be caught, and that is worth something. It cannot tell you the routing was already wrong on the day the tests were written. What you have is a regression net around a defect rather than a check against a requirement, and no dashboard on your CI distinguishes the two.
Once you know what to look for, these tests give themselves away in review. Four shapes account for nearly everything I have had to send back, and all four are written below against that same broken function.
// Shape 1 — asserts the mock was called, not what came out.
it("notifies the approver", () => {
const notify = vi.fn();
routeForApproval(50_000_000, notify);
expect(notify).toHaveBeenCalled();
// Called with what? Routed to which level? The assertion cannot tell,
// so it holds for every possible routing decision, correct or not.
});
// Shape 2 — snapshots today's output, wrong output included.
it("matches the approval snapshot", () => {
expect(approverLevel(50_000_000)).toMatchInlineSnapshot("2");
// 2 is what the code returns. The requirement says 3. The snapshot has
// just promoted the bug to the contract, and the eventual fix will look
// like a regression that needs re-recording.
});
// Shape 3 — expected value read off a run, not derived from the requirement.
it("returns level 2 at fifty million", () => {
expect(approverLevel(50_000_000)).toBe(2);
// This is the tell: the test asserts 2 because someone observed 2.
// Nothing in this file ever consulted the approval matrix.
});
// Shape 4 — every branch executed, nothing meaningful asserted in any of them.
it.each([5_000_000, 20_000_000, 80_000_000])("handles %i", (amount) => {
expect(typeof approverLevel(amount)).toBe("number");
// Three cases, all three branches, full branch coverage on the module,
// and not one statement about which level is correct.
});The property they share is that not one of them names an outcome the requirement demands. Shape 1 asserts that a call happened, shape 2 asserts the output equals itself, shape 3 asserts a number nobody derived, and shape 4 asserts a type. Replace the routing rule with any other plausible version and all four still pass. That is not a weak suite, it is a suite with no opinion at all.
Inline snapshots are the most expensive of the four, because they defend the bug actively. Once the wrong value is recorded, the correct fix makes the test fail, and the fastest route back to green is to re-record the snapshot. I have watched that happen in a review: a one-line fix, an updated snapshot in the same commit, and a green pipeline certifying the original defect.
Coverage cannot detect any of this, and not because the tools are weak. It is what the metric is defined to be. Shape 4 executes every line and both sides of every condition in that function while asserting only that the return type is a number, so it earns full marks on both counters. Three properties of the definition are worth reading slowly, because together they are the entire reason the dashboard reads well:
So a module can sit at the top of both counters with a defect on every branch, and the number is not lying. It is answering a different question from the one you are asking it. The question you actually have is whether a wrong version of this code would fail the suite, and nothing in a coverage report addresses that.
There is a measurement for the question coverage does not answer. Break the code on purpose and see whether any test fails. The tool makes one small, syntactically valid change to the source at a time, flipping a comparison, replacing a condition with true, emptying a string literal, then runs the suite against that modified copy. Each modified copy is a mutant. If a test fails, the mutant is killed, and you now have proof that some assertion depends on that line being right. If every test passes, the mutant survived.
// stryker.config.json — scope it to one module, never the whole repo
{
"packageManager": "npm",
"testRunner": "vitest",
"mutate": ["src/approval/**/*.ts"],
"reporters": ["clear-text", "progress", "html"],
"incremental": true
}
// npx stryker run
//
// Stryker rewrites the module once per mutant and re-runs the suite. Two of
// the mutants it generates for approverLevel, from its ConditionalExpression
// and EqualityOperator mutators:
//
// - if (amount > 50_000_000) return 3;
// + if (true) return 3;
//
// - if (amount > 10_000_000) return 2;
// + if (amount >= 10_000_000) return 2;
//
// Against the four tests above, both SURVIVE — Stryker's definition of
// survived is that all tests passed while the mutant was active. Line and
// branch coverage on this file read the same the whole time. The second
// mutant is the interesting one: it is the boundary bug injected on the
// other branch, and the suite has no opinion about that either.Survived is the state that matters, and Stryker's documentation is careful to separate it from the state it calls no coverage. A no-coverage mutant was never executed by any test. A survived mutant was executed by tests that did not care about the result, and that is precisely what an agent produces at scale. The headline number, the mutation score, is detected mutants over valid mutants, and unlike coverage it cannot be raised by executing more code.
Every ecosystem that matters has a mature one, and they are all a single command away from a project that already has tests. This is what I reach for per stack.
| Stack | Tool | Where to start |
|---|---|---|
| JavaScript and TypeScript | StrykerJS | npx stryker run, scoped by the mutate array in stryker.config.json |
| Java and the JVM | PIT, also called pitest | mvn test-compile org.pitest:pitest-maven:mutationCoverage |
| Python | mutmut, or cosmic-ray | mutmut run, then mutmut browse to walk the survivors |
| C# and Scala | Stryker.NET and Stryker4s | They share the StrykerJS mutator set, so the shapes above transfer unchanged |
None of these needs a new test framework. StrykerJS drives your existing Vitest or Jest run, PIT drives your existing JUnit run, and mutmut drives your existing pytest run. Trying it on one module costs one command and one wait, not a migration, which is why it is worth doing before you argue about adopting it.
Read the report from the bottom, not the top. The score is a summary you will forget; the survivor list is a set of sentences, each one saying that this specific line can be broken and your suite will ship it anyway. On the approval module, the survivors were the two comparison operators and nothing else, which told me exactly where the forty-one tests had been looking.
Mutation testing is slow, and pretending otherwise is how a team tries it once and drops it. The work is multiplicative: every mutant means another run of the tests that touch it. You can read the size of that problem off the tools themselves. PIT's own front page sells speed, saying it can analyse in minutes what earlier systems took days over. mutmut advertises remembering work already done so you can go incrementally, and knowing which tests to execute. StrykerJS ships an incremental mode that stores results in a file to speed up the next run. Take all three as evidence of the cost, not its absence. The order I run it in:
That is a risk trade made with open eyes. The modules I do not mutate are the modules where I have accepted coverage as the only signal, and being explicit about which ones those are is worth more than a repository-wide score nobody trusts. It also keeps the run inside the time budget that decides whether the practice is still alive next quarter.
Everything above is detection. The cause is one line of the prompt. If the only artefact you hand an agent is the file, the file is the only place its expected values can come from. Give it the requirement and withhold the implementation, and the tests have somewhere else to be derived from. Labelled before and after, the difference is smaller than it looks:
# BEFORE — reliably produces tests that restate the code
Read src/approval/level.ts and write unit tests for it.
Get statement coverage on that file above 90 percent.
# AFTER — produces tests that can fail
Do not open src/approval/level.ts.
Here is the requirement:
level 1 up to and including 10,000,000
level 2 above 10,000,000 up to and including 50,000,000
level 3 above 50,000,000
a negative amount is rejected, not routed
Write the tests from this requirement alone.
Include the exact boundary for every rule: 10,000,000 and 50,000,000 must
each appear as their own case, with the level the requirement gives them.
Then, before you finish: write out two plausible wrong implementations of
this rule — one that uses strict comparison at both boundaries, one that
checks the branches in the opposite order — and confirm that at least one
of your tests fails against each. Report any test that passes against both.The final instruction changes the output more than any of the others, and it is a cheap approximation of what mutation testing does properly. Make the agent describe a wrong implementation and check its own tests against it. A test that passes against both the right and the wrong version is a surviving mutant, found before the code exists instead of in a nightly report. Where there is no written specification, I write the rule into the prompt as three or four lines of prose first, and about half the time that exercise is where the real ambiguity surfaces.
When there is no specification to hand, do not go back to the file. Ask the agent to state, in prose, the rule it believes the code implements, then check that statement against your own understanding. If you disagree, you have found a bug before writing a single test. If you agree, you now have the specification you were missing and the tests can be written from it.
Coverage is not a lie. It is an accurate answer to a question that has quietly stopped being useful. When code was expensive to write, running every line was a fair proxy for someone having thought about every line, and an agent breaks that proxy by executing everything while nobody thinks about anything. The rule I keep now is short: a test earns its runtime only if it would fail against a plausible wrong implementation. Give the agent the requirement, and let a surviving mutant tell you whether it listened.
Sources and further reading