AI-Generated Tests: Coverage Rose, Mutation Score Did Not

The module was an approval router in an ERP, its coverage number had been embarrassing for a year, and I asked an agent to fix that. It read the file, wrote forty-one tests, and every single one passed on the first run. Two weeks later an amount of exactly fifty million routed to the wrong approver, at a boundary three of those new tests exercised by name.

The tests were not sloppy. They were circular, which is a different and much quieter failure: a test derived from the implementation cannot detect that the implementation is wrong, because it has already recorded the wrong answer as the expected one. This post is the mechanism behind that, the four shapes it takes in real code, the tool that catches it, what that tool honestly costs to run, and the one instruction in the prompt that stops the whole pattern.

A test written from the code cannot judge the code

Ask an agent to test a file and it will read the file. That single fact produces everything else. The expected values in the tests it hands back are the values the code produces, because the code is the only source of truth it was given, so the suite is green by construction. It would have been green against a different implementation too, as long as that implementation was the one it read.

// src/approval/level.ts
//
// Requirement, from the approval matrix in the spec:
//   level 1 up to and including 10,000,000
//   level 2 above 10,000,000 up to and including 50,000,000
//   level 3 above 50,000,000
//
// The implementation below is WRONG at both boundaries: it uses > where the
// requirement says "up to and including", so an amount of exactly 50,000,000
// routes to level 2 instead of level 3. One character per branch.
export function approverLevel(amount: number): number {
  if (amount > 50_000_000) return 3;
  if (amount > 10_000_000) return 2;
  return 1;
}

Notice what such a suite can and cannot do. It can pin current behaviour, so a later refactor that changes the routing will be caught, and that is worth something. It cannot tell you the routing was already wrong on the day the tests were written. What you have is a regression net around a defect rather than a check against a requirement, and no dashboard on your CI distinguishes the two.

Four shapes that pass whatever the code does

Once you know what to look for, these tests give themselves away in review. Four shapes account for nearly everything I have had to send back, and all four are written below against that same broken function.

// Shape 1 — asserts the mock was called, not what came out.
it("notifies the approver", () => {
  const notify = vi.fn();
  routeForApproval(50_000_000, notify);
  expect(notify).toHaveBeenCalled();
  // Called with what? Routed to which level? The assertion cannot tell,
  // so it holds for every possible routing decision, correct or not.
});

// Shape 2 — snapshots today's output, wrong output included.
it("matches the approval snapshot", () => {
  expect(approverLevel(50_000_000)).toMatchInlineSnapshot("2");
  // 2 is what the code returns. The requirement says 3. The snapshot has
  // just promoted the bug to the contract, and the eventual fix will look
  // like a regression that needs re-recording.
});

// Shape 3 — expected value read off a run, not derived from the requirement.
it("returns level 2 at fifty million", () => {
  expect(approverLevel(50_000_000)).toBe(2);
  // This is the tell: the test asserts 2 because someone observed 2.
  // Nothing in this file ever consulted the approval matrix.
});

// Shape 4 — every branch executed, nothing meaningful asserted in any of them.
it.each([5_000_000, 20_000_000, 80_000_000])("handles %i", (amount) => {
  expect(typeof approverLevel(amount)).toBe("number");
  // Three cases, all three branches, full branch coverage on the module,
  // and not one statement about which level is correct.
});

The property they share is that not one of them names an outcome the requirement demands. Shape 1 asserts that a call happened, shape 2 asserts the output equals itself, shape 3 asserts a number nobody derived, and shape 4 asserts a type. Replace the routing rule with any other plausible version and all four still pass. That is not a weak suite, it is a suite with no opinion at all.

Coverage counts execution, not verification

Coverage cannot detect any of this, and not because the tools are weak. It is what the metric is defined to be. Shape 4 executes every line and both sides of every condition in that function while asserting only that the return type is a number, so it earns full marks on both counters. Three properties of the definition are worth reading slowly, because together they are the entire reason the dashboard reads well:

• Statement coverage answers whether a line executed during the run. It has no access to the assertions in the test that executed it, and cannot be moved by them.
• Branch coverage answers whether both sides of a condition were taken somewhere in the suite. Nothing requires any single test to have distinguished those two paths from each other.
• Neither counter can tell a test that asserts the correct approval level from one that asserts the result is a number, because the assertion is not an input to either calculation.

So a module can sit at the top of both counters with a defect on every branch, and the number is not lying. It is answering a different question from the one you are asking it. The question you actually have is whether a wrong version of this code would fail the suite, and nothing in a coverage report addresses that.

Mutation testing: break the code, see who notices

There is a measurement for the question coverage does not answer. Break the code on purpose and see whether any test fails. The tool makes one small, syntactically valid change to the source at a time, flipping a comparison, replacing a condition with true, emptying a string literal, then runs the suite against that modified copy. Each modified copy is a mutant. If a test fails, the mutant is killed, and you now have proof that some assertion depends on that line being right. If every test passes, the mutant survived.

// stryker.config.json — scope it to one module, never the whole repo
{
  "packageManager": "npm",
  "testRunner": "vitest",
  "mutate": ["src/approval/**/*.ts"],
  "reporters": ["clear-text", "progress", "html"],
  "incremental": true
}

// npx stryker run
//
// Stryker rewrites the module once per mutant and re-runs the suite. Two of
// the mutants it generates for approverLevel, from its ConditionalExpression
// and EqualityOperator mutators:
//
//   - if (amount > 50_000_000) return 3;
//   + if (true) return 3;
//
//   - if (amount > 10_000_000) return 2;
//   + if (amount >= 10_000_000) return 2;
//
// Against the four tests above, both SURVIVE — Stryker's definition of
// survived is that all tests passed while the mutant was active. Line and
// branch coverage on this file read the same the whole time. The second
// mutant is the interesting one: it is the boundary bug injected on the
// other branch, and the suite has no opinion about that either.

Survived is the state that matters, and Stryker's documentation is careful to separate it from the state it calls no coverage. A no-coverage mutant was never executed by any test. A survived mutant was executed by tests that did not care about the result, and that is precisely what an agent produces at scale. The headline number, the mutation score, is detected mutants over valid mutants, and unlike coverage it cannot be raised by executing more code.

The tools, and where to start with each

Every ecosystem that matters has a mature one, and they are all a single command away from a project that already has tests. This is what I reach for per stack.

Read the report from the bottom, not the top. The score is a summary you will forget; the survivor list is a set of sentences, each one saying that this specific line can be broken and your suite will ship it anyway. On the approval module, the survivors were the two comparison operators and nothing else, which told me exactly where the forty-one tests had been looking.

The honest cost, and where to spend it

Mutation testing is slow, and pretending otherwise is how a team tries it once and drops it. The work is multiplicative: every mutant means another run of the tests that touch it. You can read the size of that problem off the tools themselves. PIT's own front page sells speed, saying it can analyse in minutes what earlier systems took days over. mutmut advertises remembering work already done so you can go incrementally, and knowing which tests to execute. StrykerJS ships an incremental mode that stores results in a file to speed up the next run. Take all three as evidence of the cost, not its absence. The order I run it in:

1. Pick the modules where a wrong answer costs money or trust. Pricing, tax, approval thresholds, permissions, anything that computes an amount. Not the whole repository, and not the UI.
2. Scope the run to exactly those paths and let it finish once. PIT takes targetClasses and targetTests globs, mutmut takes source_paths, StrykerJS takes the mutate array. Now you know the real wall-clock cost before anyone schedules it.
3. Read the survivors and fix the tests, not the score. A survivor you decide to accept should be accepted in writing, with the reason, in the same place as the rest of your test documentation.
4. Only then wire it into CI, and nightly rather than per-commit unless you have measured that it fits inside the wait your team will actually tolerate.

That is a risk trade made with open eyes. The modules I do not mutate are the modules where I have accepted coverage as the only signal, and being explicit about which ones those are is worth more than a repository-wide score nobody trusts. It also keeps the run inside the time budget that decides whether the practice is still alive next quarter.

Prompt from the specification, not the implementation

Everything above is detection. The cause is one line of the prompt. If the only artefact you hand an agent is the file, the file is the only place its expected values can come from. Give it the requirement and withhold the implementation, and the tests have somewhere else to be derived from. Labelled before and after, the difference is smaller than it looks:

# BEFORE — reliably produces tests that restate the code
Read src/approval/level.ts and write unit tests for it.
Get statement coverage on that file above 90 percent.

# AFTER — produces tests that can fail
Do not open src/approval/level.ts.

Here is the requirement:
  level 1 up to and including 10,000,000
  level 2 above 10,000,000 up to and including 50,000,000
  level 3 above 50,000,000
  a negative amount is rejected, not routed

Write the tests from this requirement alone.
Include the exact boundary for every rule: 10,000,000 and 50,000,000 must
each appear as their own case, with the level the requirement gives them.

Then, before you finish: write out two plausible wrong implementations of
this rule — one that uses strict comparison at both boundaries, one that
checks the branches in the opposite order — and confirm that at least one
of your tests fails against each. Report any test that passes against both.

The final instruction changes the output more than any of the others, and it is a cheap approximation of what mutation testing does properly. Make the agent describe a wrong implementation and check its own tests against it. A test that passes against both the right and the wrong version is a surviving mutant, found before the code exists instead of in a nightly report. Where there is no written specification, I write the rule into the prompt as three or four lines of prose first, and about half the time that exercise is where the real ambiguity surfaces.

Coverage is not a lie. It is an accurate answer to a question that has quietly stopped being useful. When code was expensive to write, running every line was a fair proxy for someone having thought about every line, and an agent breaks that proxy by executing everything while nobody thinks about anything. The rule I keep now is short: a test earns its runtime only if it would fail against a plausible wrong implementation. Give the agent the requirement, and let a surviving mutant tell you whether it listened.