# Prompt Regression Testing in CI: Assert a Rate, Not an Answer

> A prompt has no type system and no compiler, so a suite of tests is the only thing standing between a reworded instruction and a changed product. Here is how I build that suite: cases lifted from real incidents, deterministic assertions before any judge, a judge calibrated against my own labels, a pinned model, and the three things a red run can mean.

URL: https://www.matthewswong.com/en/blog/prompt-regression-testing-ci-evals/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-09-15
- Updated: 2026-09-15
- Category: AI

> **Key takeaway:** Prompt regression testing means running a fixed set of real production failures against a pinned model version on every prompt change. Assert deterministically first — valid JSON, required fields, refusals — and sample each case several times, because identical calls do not return identical text. Reach for an LLM judge only after calibrating it against your own labels.

The change was one sentence. A supplier-invoice extractor had a system prompt whose last paragraph was about output format, and I moved a line asking for brevity to the top of it, because the responses were padded. Every document I tried by hand came back as clean JSON. Four days later a batch job died on a scanned invoice, and the reason was that the model had taken to writing one courteous sentence before the JSON on inputs it found unusual.

Nothing in that pull request was reviewable. A prompt is code with no type system and no compiler, so the only thing between a reworded instruction and a changed product is a suite that runs the prompt and checks what comes back. This post is how I build that suite and, more importantly, how I keep it honest: where the cases come from, why the cheap assertions do most of the work, how to calibrate a judge before trusting one, and what to do on the morning it is red.

## Every fixture is a bug that already happened

Start the fixture set from the incident queue, not from a whiteboard. A case you invent tests behaviour you already believe works. A case lifted from a support ticket tests behaviour that has failed once, in front of a real user, which is the only hard evidence you have that this prompt can fail at all. It is also what stops the suite becoming a museum of things that were never at risk, and it means the suite grows for a reason instead of on a schedule.

Two habits keep those cases valuable. Copy the offending input verbatim: the stray thousands separator, the trailing dash after an amount, the line break in the middle of a supplier name are the entire reason the case exists, and normalising them quietly deletes the test. And write the assertion as the complaint rather than as your idea of a good answer. The reporter said the parser threw, so the assertion is that the output parses. The reporter said the total was wrong, so the assertion names that total exactly. A provenance comment naming the ticket and the date is what lets a future engineer decide whether the case still matters.

## Deterministic assertions first, and usually only those

Most prompt regressions are shape regressions, and shape is checkable without a second model. Before any judge gets involved, four questions cover the majority of what actually breaks, and all four are ordinary code:

- Does it parse? Valid JSON against the response schema, checked with the same validator the application uses, not a looser one written for the tests.
- Does it contain what it must? The required field is present, the forbidden phrase is absent, the identifier that went in comes back out.
- Does it refuse when it should? The inputs you decided the prompt must decline are still declined. This is the layer that silently disappears when somebody rewrites the system prompt for tone.
- Is it within budget? Output length, latency and cost per call all have caps, and an edit that doubles the response is a regression even when every word of it is correct.

Anthropic's own guidance on building evaluations argues the same trade from the other side: prioritise volume over quality, because more questions with slightly lower-signal automated grading beat fewer questions graded by hand. Cheap assertions are what let a suite be large enough to be representative. A judged suite of twenty cases is a smaller instrument than a code-graded suite of four hundred, whatever the sophistication of the grading.

> promptfoo ships these as named assertion types rather than leaving you to write them: is-json with a schema, contains-all, icontains-any, is-refusal, starts-with, word-count, latency and cost, plus javascript and python escape hatches, and a not- prefix that negates any of them. Its YAML is a sensible place to start even if you eventually run your own harness.

## The same prompt does not return the same output

A single run proves almost nothing, and it is worth knowing exactly why. Thinking Machines Lab sampled one prompt a thousand times at temperature 0 against Qwen3-235B and got eighty unique completions, identical for the first 102 tokens and diverging after that. The cause they identify is not the floating-point folklore usually offered but a lack of batch invariance in inference kernels: the result depends on the batch a request lands in, and the batch depends on how busy the server was, which is not yours to control.

So sample. Every case runs several times and the result is a count, not a boolean. I split cases into two tiers, contract rules that must hold on every sample and quality rules that may miss one, because a suite with a single threshold either tolerates broken JSON or fails on a synonym. Turning the temperature down is not the escape it looks like either: Anthropic now deprecates temperature, top_p and top_k on Claude Opus 4.7 and later, where a non-default value returns a 400. The parameter people reach for to make evals reproducible is itself being retired.

*Only the bottom band needs a second model call, and it is the one most teams add first. The three above it are ordinary code, and between them they catch the shape regressions that make up the bulk of what a prompt edit actually breaks.*

## A judge you have not calibrated is an unmeasured instrument

Some things genuinely need a second model: faithfulness to a source document, whether an answer is responsive at all, tone against a written rubric. The mistake is to add the judge and then believe its numbers, because at that point you have introduced a measuring device and never checked it against anything. Calibrate it the boring way. Label a sample of outputs yourself, run the judge over the same outputs, and compute the agreement.

Use Cohen's kappa rather than raw percentage agreement. Kappa is the observed agreement minus the agreement expected by chance, divided by one minus that chance agreement, which is why a judge that answers pass to everything on a sample that is ninety percent pass scores ninety percent agreement and a kappa of zero. The Landis and Koch bands from 1977 are the ones usually quoted — 0.41 to 0.60 moderate, 0.61 to 0.80 substantial — and Wikipedia is right to record that they were personal opinion and that no single kappa value is universally acceptable. Pick your own bar, write it in the repository next to the rubric, and re-measure when either moves.

> The judge is a prompt too. It drifts with everything else, so it needs its own pinned model and its own small set of regression cases, including outputs you know are bad and expect it to fail. Judging with the model that produced the output is the specific thing to avoid: independence is the property you are paying for, and you give it away for nothing.

## Pin the model, then run the suite when you move it

A suite that runs against a moving model measures two variables at once and can attribute neither. Pin the exact model identifier in one file, import it everywhere including the eval harness, and treat a change to that line as a deploy needing the same review as a prompt edit. The value of pinning is not that a pinned model is better. It is that when the suite goes red you already know which of the two things moved.

| What changed | Who initiated it | What the suite has to do |
| --- | --- | --- |
| Prompt wording | You, in a pull request | Run the full suite on the branch, with the unchanged prompt as the control |
| Pinned model identifier | You, deliberately | Run both models over the same cases and read the per-case delta before merging |
| Model retired upstream | The provider, on notice | Migrate before the date, and let the suite choose between the replacements |
| Retrieved context or tool output | Nobody, silently | Only a scheduled run against an unchanged prompt ever sees this one |

Pinning buys determinism of intent, not immortality. Anthropic's deprecation page is blunt that requests to a retired model fail, and it publishes a lifecycle — active, legacy, deprecated, retired — with a commitment of at least sixty days of notice before a publicly released model is retired. claude-3-7-sonnet-20250219 was deprecated on 28 October 2025 and retired on 19 February 2026, and applications pinned to it stopped working on that date rather than degrading. The suite is what turns the forced migration into a measured change: run it against each candidate replacement, read the per-case diff, and decide with evidence instead of a release note.

## The cheap tier on every pull request, the dear tier nightly

Splitting the suite by cost is what keeps it running at all. The deterministic tier adds no spend beyond the generation calls themselves, so it runs on every pull request that touches a prompt, the eval directory, or the file holding the pinned model identifier, and its exit code blocks the merge. The judged tier costs a second call per sample per case, so it runs nightly and on demand, and it posts a comment rather than a verdict.

Two details are cheap now and expensive later. Point the paths filter at the model-identifier file, or a model bump merges without ever running the suite that exists to guard it — this is the single most common hole in an otherwise good setup. And publish per-case results as an artefact carrying the model identifier, the sample count and the pass count for every case, because a red run is only diagnosable against the last green one, and the suite failed is not a diff.

> Record the sample count in the artefact next to the pass count. A case reading four of five today and four of five last week is stable. The same case reading four of five against a run that sampled twice is noise you have not measured yet, and the count is the whole difference between a result and an anecdote.

*The third panel is the one that costs a morning. A case that has been passing four times in five for weeks was never really passing, and reverting a good prompt because it finally landed on the wrong side of the line fixes nothing at all.*

## Three things a red suite means

Red is a question, not a verdict, and there are only three answers. Work them in this order, because the cheapest check rules out the most common mistake:

1. Re-run the failing case against the unchanged prompt, at a higher sample count. If it fails there too, the case or the assertion is the problem and your change is innocent. This is the step people skip, and skipping it is how a good prompt gets reverted.
2. Read the output, not the diff of the score. An exact-string assertion against wording that was always allowed to vary is a bad assertion, and the fix is to loosen that one assertion and record why in the case file. It is never to lower the threshold for the whole suite.
3. Check the case's history. A case that has been sitting at four of five for weeks was always marginal and has now fallen on the other side of the line. Raise its sample count until the rate means something, or retire it, but do not book noise as a regression.

One rule matters more than the three of them: never loosen a threshold in the same commit that changes a prompt. That commit is unreviewable, because it contains both the change and the permission to accept it, and it is exactly how a suite stops meaning anything to the people reading it. Loosening is its own pull request, with its own reason in the message and its own approval.

The suite is not there to prove the prompt is good. It is there so a change to the prompt produces a number two people can argue about, instead of two opinions about tone. Build it from failures that already happened, assert cheaply and often, sample enough that the number means something, calibrate anything that judges, and pin the model, so that on the morning it goes red you already know what moved.

## Sources

- Anthropic — Define success criteria and build evaluations
- Thinking Machines Lab — Defeating Nondeterminism in LLM Inference
- promptfoo — Deterministic metrics and assertion types
- Anthropic — Model deprecations, lifecycle and retirement dates
- Wikipedia — Cohen's kappa

## Source links

- https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
- https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
- https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/
- https://platform.claude.com/docs/en/about-claude/model-deprecations
- https://en.wikipedia.org/wiki/Cohen%27s_kappa

## FAQ

### What is prompt regression testing?

Prompt regression testing is running a fixed set of saved inputs through your prompt on every change and checking the outputs against assertions, the same way a unit test suite guards application code. The cases come from failures that already happened in production, and the assertions state what the reporter of each failure actually complained about. Because a prompt has no compiler, the suite is the only automated signal that a reworded instruction changed the product.

### Why does the same prompt give different answers at temperature 0?

Because inference kernels are not batch-invariant, so a request's result depends on the batch it lands in and therefore on how busy the server happens to be. Thinking Machines Lab sampled one prompt a thousand times at temperature 0 against Qwen3-235B and got eighty unique completions, identical for the first 102 tokens. This is why a single eval run proves very little and each case should be sampled several times and scored as a rate.

### Do I need an LLM judge for prompt evals?

Usually not, and never first. Deterministic assertions cover most regressions: does the output parse against the schema, does it contain the required field, does it still refuse the inputs it should refuse, is it inside the length and cost caps. Add a judge only for genuinely subjective properties like faithfulness or tone, and calibrate it against your own labels before you believe any number it produces.

### How do I know whether an LLM judge is any good?

Label a sample of outputs yourself before seeing the judge's answers, run the judge over the same outputs, and compute Cohen's kappa between the two sets of labels. Kappa corrects for agreement that would happen by chance, so a judge that passes everything on a mostly-passing sample scores zero rather than ninety percent. Pick your own acceptable band, write it down next to the rubric, and re-measure whenever the rubric or the judge model changes.

### Should I pin the model version for my prompt eval suite?

Yes. A suite running against a moving model measures two variables at once, so when it goes red you cannot tell whether your prompt edit or the provider caused it. Pin the exact model identifier in one file, import it into the eval harness too, and add that file to the CI paths filter so a model bump cannot merge without running the suite. Pinning does not prevent retirement, so plan the migration when the provider gives notice.

---

- HTML version: https://www.matthewswong.com/en/blog/prompt-regression-testing-ci-evals/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/prompt-regression-testing-ci-evals/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json