Prompt Regression Testing in CI: Assert a Rate, Not an Answer

The change was one sentence. A supplier-invoice extractor had a system prompt whose last paragraph was about output format, and I moved a line asking for brevity to the top of it, because the responses were padded. Every document I tried by hand came back as clean JSON. Four days later a batch job died on a scanned invoice, and the reason was that the model had taken to writing one courteous sentence before the JSON on inputs it found unusual.

Nothing in that pull request was reviewable. A prompt is code with no type system and no compiler, so the only thing between a reworded instruction and a changed product is a suite that runs the prompt and checks what comes back. This post is how I build that suite and, more importantly, how I keep it honest: where the cases come from, why the cheap assertions do most of the work, how to calibrate a judge before trusting one, and what to do on the morning it is red.

Every fixture is a bug that already happened

Start the fixture set from the incident queue, not from a whiteboard. A case you invent tests behaviour you already believe works. A case lifted from a support ticket tests behaviour that has failed once, in front of a real user, which is the only hard evidence you have that this prompt can fail at all. It is also what stops the suite becoming a museum of things that were never at risk, and it means the suite grows for a reason instead of on a schedule.

# evals/cases/invoice-extract/0042-amount-suffix.yaml
#
# Provenance: SUP-2411, reported 2026-02-03, closed 2026-02-05.
# A supplier sent an invoice with the total written "Rp 12.500.000,-".
# The extractor got the number right and then wrapped the JSON in a polite
# sentence, so the downstream parser threw. Both halves of that are the case.
#
# Do NOT tidy the fixture. The trailing dash after the amount and the line
# break inside the supplier name are the reason this file exists; normalising
# them turns a regression test into a formatting test.
vars:
  document: file://fixtures/SUP-2411.txt

assert:
  # The complaint, expressed literally: JSON, and nothing around it.
  - type: is-json
    value: file://schemas/invoice.schema.json

  # The number the reporter actually cared about. Not "roughly right".
  - type: javascript
    value: JSON.parse(output).total_idr === 12500000

  # The regression we shipped WHILE fixing this the first time.
  - type: not-contains
    value: "Here is"

Two habits keep those cases valuable. Copy the offending input verbatim: the stray thousands separator, the trailing dash after an amount, the line break in the middle of a supplier name are the entire reason the case exists, and normalising them quietly deletes the test. And write the assertion as the complaint rather than as your idea of a good answer. The reporter said the parser threw, so the assertion is that the output parses. The reporter said the total was wrong, so the assertion names that total exactly. A provenance comment naming the ticket and the date is what lets a future engineer decide whether the case still matters.

Deterministic assertions first, and usually only those

Most prompt regressions are shape regressions, and shape is checkable without a second model. Before any judge gets involved, four questions cover the majority of what actually breaks, and all four are ordinary code:

• Does it parse? Valid JSON against the response schema, checked with the same validator the application uses, not a looser one written for the tests.
• Does it contain what it must? The required field is present, the forbidden phrase is absent, the identifier that went in comes back out.
• Does it refuse when it should? The inputs you decided the prompt must decline are still declined. This is the layer that silently disappears when somebody rewrites the system prompt for tone.
• Is it within budget? Output length, latency and cost per call all have caps, and an edit that doubles the response is a regression even when every word of it is correct.

Anthropic's own guidance on building evaluations argues the same trade from the other side: prioritise volume over quality, because more questions with slightly lower-signal automated grading beat fewer questions graded by hand. Cheap assertions are what let a suite be large enough to be representative. A judged suite of twenty cases is a smaller instrument than a code-graded suite of four hundred, whatever the sophistication of the grading.

The same prompt does not return the same output

A single run proves almost nothing, and it is worth knowing exactly why. Thinking Machines Lab sampled one prompt a thousand times at temperature 0 against Qwen3-235B and got eighty unique completions, identical for the first 102 tokens and diverging after that. The cause they identify is not the floating-point folklore usually offered but a lack of batch invariance in inference kernels: the result depends on the batch a request lands in, and the batch depends on how busy the server was, which is not yours to control.

// evals/run.ts
//
// Two tiers, because two kinds of rule share one suite:
//   contract  valid JSON, required field, refusal  -> every sample must pass
//   quality   tone, completeness, judged           -> one sample may miss
const SAMPLES = 5;

export async function runCase(c: EvalCase): Promise<CaseResult> {
  const outputs: string[] = [];
  for (let i = 0; i < SAMPLES; i++) {
    outputs.push(
      await callModel({
        // Imported from src/llm/model.ts. Never an alias, never "latest",
        // and never read from an environment variable that CI can drift.
        model: PINNED_MODEL,
        system: c.prompt,
        user: c.input,
      }),
    );
  }

  const passes = outputs.filter((o) => c.assertions.every((a) => a(o))).length;
  const required = c.tier === "contract" ? SAMPLES : SAMPLES - 1;

  return {
    id: c.id,
    // The result is the count. "Passed" is what you derive from it, and it is
    // the count, not the boolean, that a future red run gets read against.
    passes,
    samples: SAMPLES,
    required,
    ok: passes >= required,
    model: PINNED_MODEL,
  };
}

So sample. Every case runs several times and the result is a count, not a boolean. I split cases into two tiers, contract rules that must hold on every sample and quality rules that may miss one, because a suite with a single threshold either tolerates broken JSON or fails on a synonym. Turning the temperature down is not the escape it looks like either: Anthropic now deprecates temperature, top_p and top_k on Claude Opus 4.7 and later, where a non-default value returns a 400. The parameter people reach for to make evals reproducible is itself being retired.

A judge you have not calibrated is an unmeasured instrument

Some things genuinely need a second model: faithfulness to a source document, whether an answer is responsive at all, tone against a written rubric. The mistake is to add the judge and then believe its numbers, because at that point you have introduced a measuring device and never checked it against anything. Calibrate it the boring way. Label a sample of outputs yourself, run the judge over the same outputs, and compute the agreement.

// evals/calibrate-judge.ts
//
// Cohen's kappa between MY labels and the judge's, over the same outputs.
//
//   kappa = (po - pe) / (1 - pe)
//   po = agreement observed, pe = agreement expected by chance
//
// Why not plain percentage agreement: on a sample where 90 percent of
// outputs are good, a judge that answers "good" to everything scores
// 90 percent and has measured nothing at all. Kappa scores it 0.
type Label = "pass" | "fail";

export function cohensKappa(mine: Label[], judge: Label[]): number {
  const n = mine.length;
  const agreed = mine.filter((m, i) => m === judge[i]).length;
  const po = agreed / n;

  const rate = (xs: Label[], l: Label) => xs.filter((x) => x === l).length / n;
  const pe =
    rate(mine, "pass") * rate(judge, "pass") +
    rate(mine, "fail") * rate(judge, "fail");

  return (po - pe) / (1 - pe);
}

// Label the sample BEFORE you see the judge's answers, and re-run this
// whenever the rubric, the judge prompt or the judge model moves. A judge
// that has never been through it is an instrument with no scale on it.

Use Cohen's kappa rather than raw percentage agreement. Kappa is the observed agreement minus the agreement expected by chance, divided by one minus that chance agreement, which is why a judge that answers pass to everything on a sample that is ninety percent pass scores ninety percent agreement and a kappa of zero. The Landis and Koch bands from 1977 are the ones usually quoted — 0.41 to 0.60 moderate, 0.61 to 0.80 substantial — and Wikipedia is right to record that they were personal opinion and that no single kappa value is universally acceptable. Pick your own bar, write it in the repository next to the rubric, and re-measure when either moves.

Pin the model, then run the suite when you move it

A suite that runs against a moving model measures two variables at once and can attribute neither. Pin the exact model identifier in one file, import it everywhere including the eval harness, and treat a change to that line as a deploy needing the same review as a prompt edit. The value of pinning is not that a pinned model is better. It is that when the suite goes red you already know which of the two things moved.

Pinning buys determinism of intent, not immortality. Anthropic's deprecation page is blunt that requests to a retired model fail, and it publishes a lifecycle — active, legacy, deprecated, retired — with a commitment of at least sixty days of notice before a publicly released model is retired. claude-3-7-sonnet-20250219 was deprecated on 28 October 2025 and retired on 19 February 2026, and applications pinned to it stopped working on that date rather than degrading. The suite is what turns the forced migration into a measured change: run it against each candidate replacement, read the per-case diff, and decide with evidence instead of a release note.

The cheap tier on every pull request, the dear tier nightly

Splitting the suite by cost is what keeps it running at all. The deterministic tier adds no spend beyond the generation calls themselves, so it runs on every pull request that touches a prompt, the eval directory, or the file holding the pinned model identifier, and its exit code blocks the merge. The judged tier costs a second call per sample per case, so it runs nightly and on demand, and it posts a comment rather than a verdict.

# .github/workflows/prompt-evals.yml
name: prompt-evals

on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      # The pinned model ID lives here. Leave this line out and a model bump
      # merges without ever running the suite that exists to guard it.
      - "src/llm/model.ts"
  schedule:
    # Nightly, against an unchanged prompt. This run is the only thing that
    # sees drift nobody on the team initiated.
    - cron: "0 19 * * *"
  workflow_dispatch:

jobs:
  contract:
    # Deterministic assertions only. Fast, no judge, blocks the merge.
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx tsx evals/run.ts --tier contract --samples 5
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results-contract
          # Per case: model ID, sample count, pass count. A red run is only
          # diagnosable against the last green one.
          path: evals/out/results.json

  judged:
    # Costs a second model call per sample. Nightly and on demand only,
    # and it comments rather than blocking.
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx tsx evals/run.ts --tier quality --samples 5 --judge

Two details are cheap now and expensive later. Point the paths filter at the model-identifier file, or a model bump merges without ever running the suite that exists to guard it — this is the single most common hole in an otherwise good setup. And publish per-case results as an artefact carrying the model identifier, the sample count and the pass count for every case, because a red run is only diagnosable against the last green one, and the suite failed is not a diff.

Three things a red suite means

Red is a question, not a verdict, and there are only three answers. Work them in this order, because the cheapest check rules out the most common mistake:

1. Re-run the failing case against the unchanged prompt, at a higher sample count. If it fails there too, the case or the assertion is the problem and your change is innocent. This is the step people skip, and skipping it is how a good prompt gets reverted.
2. Read the output, not the diff of the score. An exact-string assertion against wording that was always allowed to vary is a bad assertion, and the fix is to loosen that one assertion and record why in the case file. It is never to lower the threshold for the whole suite.
3. Check the case's history. A case that has been sitting at four of five for weeks was always marginal and has now fallen on the other side of the line. Raise its sample count until the rate means something, or retire it, but do not book noise as a regression.

One rule matters more than the three of them: never loosen a threshold in the same commit that changes a prompt. That commit is unreviewable, because it contains both the change and the permission to accept it, and it is exactly how a suite stops meaning anything to the people reading it. Loosening is its own pull request, with its own reason in the message and its own approval.

The suite is not there to prove the prompt is good. It is there so a change to the prompt produces a number two people can argue about, instead of two opinions about tone. Build it from failures that already happened, assert cheaply and often, sample enough that the number means something, calibrate anything that judges, and pin the model, so that on the morning it goes red you already know what moved.