LLM Evals in CI: Testing Prompts Like Code

The first time a prompt change silently broke production for me, nobody noticed for four days. A support-triage assistant had been rewritten to sound friendlier, and in the process it stopped reliably tagging billing complaints as billing complaints. The change passed code review because there was no code to review, only a paragraph of English. That is the core problem with prompts: they are executable logic that ships with none of the safety nets we would insist on for a function doing the same job.

The fix is not exotic. It is the same discipline we already apply to application code, adapted to a system whose output is probabilistic rather than deterministic. This post walks through building an evaluation harness that runs in continuous integration: a golden dataset of representative cases, a mix of code-based assertions and LLM-judge scoring, a pass-rate gate that blocks merges, and a workflow for catching drift before a customer does.

Why prompts need the same gate as code

A prompt is a specification with fuzzy edges. Two prompts that look nearly identical in wording can produce meaningfully different distributions of outputs once you run them across hundreds of real inputs. Manual spot-checking in a playground only samples a handful of cases, and it is exactly the cases you did not think to check that break in production.

• Prompt edits, model version bumps, and retrieval or tool configuration changes are all deploys, even if no application code changed.
• Without a fixed dataset and threshold, quality regressions are judged by vibes, and vibes do not show up in a pull request diff.
• A model provider silently updating a model behind an API alias can shift behavior with zero code changes on your side at all.
• Treating evals as a CI gate turns "does this still work" from a debate into a number everyone can agree on.

Building the golden dataset

A golden dataset is a fixed set of input and expected-output pairs that represents the behavior you are committed to preserving. Keep the first version small and focused, on the order of fifty to a hundred cases, covering the categories your feature actually handles plus the edge cases that have already bitten you: ambiguous phrasing, unusually long inputs, adversarial or hostile input, and inputs with no clean answer at all. Store it in your repository as versioned JSONL so every change to the dataset shows up in a diff, exactly like a change to test fixtures would.

# golden/support-triage.jsonl
{"input": "My invoice shows double billing for March", "expected_category": "billing", "expected_contains": ["refund", "billing team"]}
{"input": "The API returns 500 on every /v2/export call", "expected_category": "bug", "expected_contains": ["engineering", "ticket"]}
{"input": "How do I add a teammate to my workspace?", "expected_category": "how_to", "expected_contains": ["settings", "invite"]}

# eval_runner.py (simplified)
import json, subprocess

def run_case(case, model_fn):
    output = model_fn(case["input"])
    checks = {
        "category_match": case["expected_category"] in output.lower(),
        "contains_terms": all(term in output.lower() for term in case["expected_contains"]),
    }
    return all(checks.values()), checks

def run_suite(path, model_fn):
    passed, total = 0, 0
    for line in open(path):
        case = json.loads(line)
        ok, _ = run_case(case, model_fn)
        passed += ok
        total += 1
    return passed / total

Each case needs an assertion strategy, not just an expected string. For classification-style tasks you can check for an exact category match. For open-ended generation you check for required substrings, forbidden substrings, or structural properties like valid JSON. The runner below loads the dataset, calls the model function under test, and reports a pass rate you can compare against a threshold in CI.

Two grading strategies, and when to use each

Not every output can be graded with a string comparison, and not every output needs the expense of a second model call to judge it. Most mature eval suites blend both approaches, choosing per case rather than per suite.

When you do reach for an LLM judge, use a different model than the one generating the output under test, and give the judge a rubric with concrete anchors rather than an open-ended "rate this response" prompt. A rubric that says what a one and what a five look like produces far more consistent scores across runs than a bare scale.

Setting a pass-rate threshold that means something

A raw pass rate is only useful once you have decided what counts as acceptable, and that decision should come from your current baseline, not from an arbitrary round number. The process that has worked well for me:

1. Run the full golden set against the current production prompt and record the pass rate as the baseline.
2. Set the CI threshold slightly below that baseline, leaving room for the noise inherent in any model call, rather than demanding a perfect one hundred percent that would fail on unrelated flakiness.
3. Re-baseline deliberately, with a reviewed pull request, whenever you intentionally accept a new tradeoff, so the threshold always reflects a decision someone made on purpose.

Cases that consistently pass across many prompt iterations are candidates to graduate into a stricter regression suite that must stay near a hundred percent, while newer or harder cases can live in a separate suite you are still actively improving against.

Wiring the gate into CI

The workflow below triggers only when prompts, the eval runner, or the golden dataset itself change, so routine application code changes are not paying for a model-call-heavy job on every push. It runs the code-graded suite and the judge-scored suite as separate steps so a failure in one does not obscure the other's result.

# .github/workflows/llm-eval.yml
name: LLM Eval Gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "eval_runner.py"
      - "golden/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Run golden set eval
        run: python eval_runner.py --suite golden/support-triage.jsonl --min-pass-rate 0.92
      - name: Run judge-scored eval
        run: python judge_eval.py --suite golden/tone.jsonl --min-avg-score 4.0

Keep the eval job's exit code the source of truth for the merge decision, and publish the per-case results as a build artifact or PR comment so a reviewer can see exactly which cases regressed, not just that the aggregate score dropped. A single number without the underlying cases is a much harder regression to debug at 5pm on a Friday.

What this catches that manual testing misses

The value of an automated eval suite is not that it is smarter than a careful engineer manually testing a change. It is that it runs the exact same fixed set of cases every single time, with no fatigue, no forgetting the weird edge case from three months ago, and no skipping the check because the change looked small.

• Silent model provider updates behind a stable API alias, caught the next time CI runs even without a code change on your side.
• Prompt edits that improve the common case while quietly breaking a less common but still important category, visible immediately as a per-category pass rate drop.
• Drift introduced by upstream changes to retrieved context or tool outputs, since the golden set exercises the whole pipeline, not the prompt text in isolation.

Start small. A fifty-case golden set with a simple pass-rate gate catches far more regressions than no eval suite at all, and it is a foundation you can grow into judge-scored rubrics and per-category thresholds as the feature matures.