Evals: testing non-deterministic systems

Traditional software testing assumes a function returns the same output for the same input, and that you can write down what the right output is. LLM features break both assumptions. The same prompt can produce different answers, many different answers can be acceptable, and a one-word prompt edit or a model upgrade can quietly change behaviour across thousands of cases. Evaluation — “evals” — is how you regain the ability to know whether your system works. It is not a final QA step; it is the core discipline that makes every other improvement possible.

Why evals come first

Without evals, development runs on vibes: try a prompt on three examples, it looks better, ship it. Then something that used to work breaks, and nobody notices until a user does. Hamel Husain puts it bluntly in Your AI Product Needs Evals: the unsuccessful AI products he has seen tend to share a failure to build robust evaluation systems.

Evals give you:

• A way to tell a real improvement from a lucky sample.
• Confidence to change prompts, models and retrieval without fear.
• A shared definition of “good” that a team can argue about concretely.

Start by looking at the data

Before writing any grader, read outputs. Take a few dozen real or realistic inputs, run them, and look at every response. Write down what goes wrong in plain words: “ignores the date range”, “invents a policy”, “too long for mobile”, “wrong language”.

You cannot measure a failure mode you have not noticed yet.

This error analysis tells you what to measure. Group the notes into categories and count them; the most frequent or most harmful categories are where your first checks should go. Repeat regularly — new failure modes appear as usage grows.

Build a representative test set

A good test set looks like your real traffic, not like the examples you thought of on day one.

• Pull from production logs where you can (respecting privacy).
• Add every failure you find, so fixes are verified and cannot silently regress.
• Include edge cases deliberately: empty input, very long input, ambiguous requests, adversarial text, other languages.
• Record expectations: a reference answer, required facts, or the properties a good answer must have.

Small is fine to start. Fifty well-chosen cases beat a thousand synthetic ones nobody has read. Synthetic data generated by a model can help fill gaps, but check it against reality.

Three kinds of graders

Code-graded

Anything objectively checkable should be checked in code: does the output parse as JSON, match a schema, include required fields, stay under a length, reference a source ID that exists, avoid a banned phrase, pick the correct label? These checks are fast, cheap and perfectly consistent, and they often catch a surprising share of real failures.

def check(output: str, case) -> bool:
    data = json.loads(output)          # fails if not JSON
    return data["category"] == case.expected_category

Model-graded (LLM-as-judge)

Qualities like helpfulness, tone, or whether an answer is supported by its sources are hard to check with code. Here you can ask a model to grade, given a rubric. Zheng et al. (2023) found that a strong model judge could reach agreement with human preferences comparable to the agreement between humans — but they also documented systematic biases:

• Position bias — favouring an answer because of where it appears in a comparison.
• Verbosity bias — favouring longer answers.
• Self-enhancement bias — favouring outputs resembling the judge’s own.
• Weakness on maths and reasoning — judges can be misled by plausible but wrong work.

Practical mitigations: write narrow, specific criteria; prefer pass/fail judgments over vague 1–10 scales; evaluate pairwise comparisons in both orders; give the judge reference answers when you have them. Most importantly, validate the judge: label a sample by hand and measure how often the judge agrees with you. An unvalidated judge is just another unverified model output.

Human evaluation

People remain the ground truth for subjective or high-stakes quality, and the only way to calibrate automated graders. Human review is slow and costly, so spend it where it matters: labelling calibration sets, reviewing ambiguous cases, and periodically auditing what automated checks are passing.

Regression testing

Every change to a prompt, model version, retrieval setup, or sampling parameter should rerun the suite. Compare against the previous version at the case level, not just the average. An overall score can rise while important cases newly fail.

Non-determinism means some care with numbers:

• Grade properties rather than exact strings.
• Run important cases several times to see how much they vary.
• Be suspicious of small differences on small sets; a one-case swing out of twenty is easily noise.

Wire the fast, cheap checks into CI so they run on every change, and run slower model-graded or human review before significant releases.

Offline and online

Offline evaluation runs on your fixed test set before you ship. It is repeatable and safe, but only as good as the set is representative.

Online evaluation measures live behaviour: explicit feedback, whether users accept or edit outputs, escalations to humans, task completion, A/B tests, and grading a sample of production traffic with the same checks you use offline. It catches the inputs you never imagined.

The two feed each other. Production failures become offline test cases; offline suites protect you from shipping regressions into production.

Honest caveats

Public benchmark scores tell you little about your feature — the tasks differ, and benchmarks can leak into training data. Metrics can be gamed by your own iteration: tune long enough against a small set and you overfit to it. And no eval suite is complete. The durable habit is the loop: look at data, name the failures, measure them, change something, measure again.