AI
LLM Evals in CI: Testing Prompts Like Code
July 202610 min read
Photo by Euphy on flickr
A golden dataset is a fixed, versioned collection of input and expected-output pairs that represents the behavior a prompt or model must preserve. It typically starts small, around fifty to a hundred cases pulled from real production failures and known edge cases, and lives in the repository so changes to it show up in a diff like any other test fixture.
A prompt is executable logic even though it has no compiler to catch mistakes, and small wording changes can shift output behavior across an entire distribution of inputs in ways manual spot-checking will miss. Running a fixed golden set on every prompt-touching pull request turns quality regression from a subjective debate into an objective pass-rate number that blocks the merge automatically.
Use code-based assertions such as exact match, substring checks, or JSON schema validation whenever the task has an objectively checkable answer, since they are fast and deterministic. Reserve an LLM judge for subjective qualities like tone, helpfulness, or faithfulness, and grade with a different model than the one under test to reduce self-preference bias.
Yes, that is one of the strongest arguments for running the golden set in CI on a schedule as well as on pull requests. If a provider updates the model behind a stable API alias, the next scheduled eval run will show a pass-rate drop even though nothing in your repository changed, giving you an early signal before users notice.
Run the current production prompt against the full golden set once to establish a baseline pass rate, then set the CI threshold slightly below that baseline to absorb the normal noise of model calls rather than demanding a brittle one hundred percent. Re-baseline deliberately through a reviewed pull request whenever you knowingly accept a new tradeoff.
Photo by Euphy on flickr
The first time a prompt change silently broke production for me, nobody noticed for four days. A support-triage assistant had been rewritten to sound friendlier, and in the process it stopped reliably tagging billing complaints as billing complaints. The change passed code review because there was no code to review, only a paragraph of English. That is the core problem with prompts: they are executable logic that ships with none of the safety nets we would insist on for a function doing the same job.
The fix is not exotic. It is the same discipline we already apply to application code, adapted to a system whose output is probabilistic rather than deterministic. This post walks through building an evaluation harness that runs in continuous integration: a golden dataset of representative cases, a mix of code-based assertions and LLM-judge scoring, a pass-rate gate that blocks merges, and a workflow for catching drift before a customer does.
A prompt is a specification with fuzzy edges. Two prompts that look nearly identical in wording can produce meaningfully different distributions of outputs once you run them across hundreds of real inputs. Manual spot-checking in a playground only samples a handful of cases, and it is exactly the cases you did not think to check that break in production.
Start your golden dataset from real production failures, not hypothetical edge cases you invent at a whiteboard. A failure that already happened once is a failure that will happen again.
A golden dataset is a fixed set of input and expected-output pairs that represents the behavior you are committed to preserving. Keep the first version small and focused, on the order of fifty to a hundred cases, covering the categories your feature actually handles plus the edge cases that have already bitten you: ambiguous phrasing, unusually long inputs, adversarial or hostile input, and inputs with no clean answer at all. Store it in your repository as versioned JSONL so every change to the dataset shows up in a diff, exactly like a change to test fixtures would.
# golden/support-triage.jsonl
{"input": "My invoice shows double billing for March", "expected_category": "billing", "expected_contains": ["refund", "billing team"]}
{"input": "The API returns 500 on every /v2/export call", "expected_category": "bug", "expected_contains": ["engineering", "ticket"]}
{"input": "How do I add a teammate to my workspace?", "expected_category": "how_to", "expected_contains": ["settings", "invite"]}
# eval_runner.py (simplified)
import json, subprocess
def run_case(case, model_fn):
output = model_fn(case["input"])
checks = {
"category_match": case["expected_category"] in output.lower(),
"contains_terms": all(term in output.lower() for term in case["expected_contains"]),
}
return all(checks.values()), checks
def run_suite(path, model_fn):
passed, total = 0, 0
for line in open(path):
case = json.loads(line)
ok, _ = run_case(case, model_fn)
passed += ok
total += 1
return passed / total
Each case needs an assertion strategy, not just an expected string. For classification-style tasks you can check for an exact category match. For open-ended generation you check for required substrings, forbidden substrings, or structural properties like valid JSON. The runner below loads the dataset, calls the model function under test, and reports a pass rate you can compare against a threshold in CI.
Not every output can be graded with a string comparison, and not every output needs the expense of a second model call to judge it. Most mature eval suites blend both approaches, choosing per case rather than per suite.
| Grading method | Best for | Tradeoff |
|---|---|---|
| Code-based assertion | Classification, required fields, valid JSON, forbidden terms | Fast and deterministic, but brittle to valid rewordings |
| LLM-as-judge | Tone, helpfulness, faithfulness to source, subjective quality | Flexible and nuanced, but non-deterministic and costs a call |
| Embedding similarity | Paraphrase consistency, semantic drift between versions | Cheap to compute, but a blunt proxy for correctness |
When you do reach for an LLM judge, use a different model than the one generating the output under test, and give the judge a rubric with concrete anchors rather than an open-ended "rate this response" prompt. A rubric that says what a one and what a five look like produces far more consistent scores across runs than a bare scale.
An LLM judge scored by the same model family that produced the output tends to rate its own sibling's answers more favorably. If your budget allows it, judge with a different provider or at minimum a different model size to reduce that self-preference bias.
A raw pass rate is only useful once you have decided what counts as acceptable, and that decision should come from your current baseline, not from an arbitrary round number. The process that has worked well for me:
Cases that consistently pass across many prompt iterations are candidates to graduate into a stricter regression suite that must stay near a hundred percent, while newer or harder cases can live in a separate suite you are still actively improving against.
The workflow below triggers only when prompts, the eval runner, or the golden dataset itself change, so routine application code changes are not paying for a model-call-heavy job on every push. It runs the code-graded suite and the judge-scored suite as separate steps so a failure in one does not obscure the other's result.
# .github/workflows/llm-eval.yml
name: LLM Eval Gate
on:
pull_request:
paths:
- "prompts/**"
- "eval_runner.py"
- "golden/**"
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- name: Run golden set eval
run: python eval_runner.py --suite golden/support-triage.jsonl --min-pass-rate 0.92
- name: Run judge-scored eval
run: python judge_eval.py --suite golden/tone.jsonl --min-avg-score 4.0
Keep the eval job's exit code the source of truth for the merge decision, and publish the per-case results as a build artifact or PR comment so a reviewer can see exactly which cases regressed, not just that the aggregate score dropped. A single number without the underlying cases is a much harder regression to debug at 5pm on a Friday.
Once this gate exists, a prompt change becomes a normal pull request: open it, watch the eval job run against the golden set, and merge once the pass rate holds. No more manually re-testing a dozen scenarios in a chat window before every release.
The value of an automated eval suite is not that it is smarter than a careful engineer manually testing a change. It is that it runs the exact same fixed set of cases every single time, with no fatigue, no forgetting the weird edge case from three months ago, and no skipping the check because the change looked small.
Start small. A fifty-case golden set with a simple pass-rate gate catches far more regressions than no eval suite at all, and it is a foundation you can grow into judge-scored rubrics and per-category thresholds as the feature matures.