Evaluating automated research output

An agent built you a pipeline: it pulls survey verbatims, sends each one to a model, and writes back a theme, a sentiment and a short quote. It runs nightly, it has never thrown an error, and the dashboard is green. None of that tells you whether the output is any good. This topic is about the missing half: measuring correctness, and noticing when it slips.

Start with a gold set, not a metric

A gold set is a fixed sample of real inputs with the right answer written down by a human. Everything else here depends on it. Without one, “is the pipeline accurate?” has no answer, only opinions.

Three properties make a gold set worth having:

• Frozen. The same items every time, so today’s score is comparable with last month’s.
• Stratified. Deliberately include the awkward cases: one-word answers, sarcasm, two brands in one sentence, a respondent answering a different question. If you sample purely at random you get a set of easy items and a flattering score.
• Honestly labelled. By someone who knows the domain, before they see what the model said.

Fifty to two hundred items is usually enough to be informative and small enough that a person will actually label it. Store it in the repo next to the pipeline:

# eval/gold/themes.jsonl — one hand-labelled item per line
{"id": "r-10482", "text": "Delivery was fine but the app kept logging me out",
 "themes": ["app_reliability"], "sentiment": "mixed"}
{"id": "r-10513", "text": "no comment", "themes": [], "sentiment": "none"}

That second line matters. Empty and junk inputs are where pipelines invent things, and a gold set without them will never catch it.

Precision and recall, in plain terms

Suppose the pipeline’s job is to flag verbatims that mention price sensitivity.

• Precision: of the ones it flagged, how many should have been flagged. The “don’t cry wolf” number.
• Recall: of the ones that should have been flagged, how many it found. The “don’t miss things” number.

They pull against each other. Loosen the criteria and recall rises while precision falls; tighten them and the reverse. Which you favour is a business decision, not a technical one. A missed adverse mention in a client deliverable is worse than a false one you delete in review, so lean towards recall and budget for human checking. If the false flags are so numerous that researchers stop reading the output, you have automated nothing.

from sklearn.metrics import precision_recall_fscore_support, confusion_matrix

y_true = [item["price_sensitive"] for item in gold]
y_pred = [run_pipeline(item["text"]) for item in gold]

# average="binary" keeps the numbers interpretable for one label
p, r, f1, _ = precision_recall_fscore_support(y_true, y_pred, average="binary")
print(f"precision={p:.2f} recall={r:.2f} f1={f1:.2f}")
print(confusion_matrix(y_true, y_pred))  # read the false negatives, not the summary

The habit that actually improves a pipeline is reading the individual disagreements. A single overall accuracy figure tells you nothing about how it is wrong. Print the twelve items it got wrong and read them.

Before you blame the model: do humans agree?

Ask two experienced researchers to code the same hundred verbatims for the same category. If they agree on only 68 of them, the category is under-defined, and no prompt can score above that ceiling. Inter-rater agreement measures this, usually as Cohen’s kappa, which discounts the agreement you would get by guessing.

from sklearn.metrics import cohen_kappa_score

# two humans, same 100 items, coded independently
kappa = cohen_kappa_score(rater_a, rater_b)
# < 0.4: the definition is the problem, not the pipeline

Low agreement is a brief to rewrite the definition, add examples and counter-examples, and rebuild the gold set. Teams routinely skip this and spend weeks tuning a prompt against a target that was never coherent.

LLM-as-judge, and how it misleads you

An LLM judge is a second model call that scores the first one’s output. It is genuinely useful: it is cheap enough to run across thousands of items, so you can compare two prompt versions at a scale no human can match, and it can triage which items a person should read.

It also has well-documented biases. The MT-Bench work found judges prefer whichever answer appears in a particular position, prefer longer and more confident-sounding answers regardless of correctness, and favour output from models like themselves. On top of that, a judge is lenient about anything it cannot check, which includes most factual claims.

A judge an agent writes usually looks like this:

# Before: one vague number, maximum bias exposure
prompt = f"Rate this summary from 1 to 10.\n\nSummary: {summary}"
score = float(call_model(prompt))

The better version constrains the judgement and controls for order:

class Verdict(BaseModel):                      # pydantic v2
    grounded: bool                             # every claim traceable to the source
    covers_key_points: bool
    invents_numbers: bool
    note: str

def judge(source: str, a: str, b: str) -> Verdict:
    # run both orderings and keep only agreeing verdicts; disagreements go to a human
    ...

Then the step almost nobody does: evaluate the judge. Run it over the gold set and measure its precision and recall against the human labels. If your judge agrees with humans 75% of the time, a judge-reported “92% pass rate” means very little, and you should report both figures.

Regression tests for prompts

A prompt is code. Edit it and you can break cases you never looked at. The reason people don’t test prompts is that exact-match assertions fail on harmless rewording, so the tests get deleted. Assert on invariants instead.

import pytest

CASES = load_jsonl("eval/gold/themes.jsonl")

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_extraction_invariants(case):
    out = extract_themes(case["text"])        # returns a validated pydantic model
    assert out.sentiment in {"positive", "negative", "mixed", "none"}
    if not case["themes"]:
        assert out.themes == []               # must not invent a theme from "no comment"
    for expected in case["themes"]:
        assert expected in out.themes         # must not lose a known theme

def test_gold_set_score_floor():
    p, r = score_against_gold(CASES)
    assert r >= 0.80, f"recall regression: {r:.2f}"   # a floor, not a target

Run these in CI against a cheap model, and keep the expensive full-gold-set run as a job you trigger before a release. The floor assertion is the one that saves you: it turns “the new prompt feels better” into a number that has to hold.

Sampling production, and watching the trend

The gold set tells you about the past. Production drifts, so sample it continuously: a small random slice of every run, plus a targeted slice of the lowest-confidence items and anything from a new supplier or document type. Random keeps the estimate unbiased; targeted finds the new failure first.

Record the verdict somewhere durable, so checks accumulate instead of evaporating:

-- one row per human-checked item; accuracy over time becomes a query
CREATE TABLE eval_checks (
  run_id      TEXT NOT NULL,
  item_id     TEXT NOT NULL,
  checked_at  TEXT NOT NULL,
  verdict     TEXT NOT NULL CHECK (verdict IN ('correct','wrong','partial')),
  reviewer    TEXT NOT NULL,
  note        TEXT
);

The failure mode: silent degradation

The pipeline that breaks loudly is the easy case. The one that matters is the pipeline that keeps returning 200s, keeps writing rows, and gets quietly worse. A model version changes under a floating alias. A supplier adds a column and your parser reads the wrong field. A PDF extractor starts truncating at page two. A colleague edits the prompt to fix one client’s complaint.

Every one of those leaves uptime, error rate and job duration untouched. So emit quality signals per run and chart them:

log.info("run_quality", extra={
    "run_id": run_id,
    "items_in": n_in,
    "items_out": n_out,
    "empty_output_rate": n_empty / n_in,       # the single most useful number
    "parse_failure_rate": n_parse_fail / n_in,
    "mean_judge_score": mean_score,
    "model": MODEL_ID,                         # pin it; log it; alert on change
    "prompt_sha": PROMPT_SHA,
})

Alert on movement in empty_output_rate and parse_failure_rate, not on a threshold you guessed. And pin the model: gpt-4o style floating aliases mean your pipeline can change overnight without a single commit.

What to check in your own code

• Is there a eval/ or tests/gold directory at all? If not, that is the first thing to build.
• Grep for the model identifier. Is it a pinned version, or a floating alias? Is it logged with each run?
• Does the code path for an empty, one-word or junk input return empty, or does it return a confident-looking answer?
• Is there any assertion on output quality in CI, or only tests that the function returns without raising?
• Find the judge, if there is one. Does it emit a single number? Is its own accuracy against human labels written down anywhere?
• Open the pipeline’s run log. Can you see items-in and items-out per run, or only “completed successfully”?
• Was the gold set used to tune the prompt? If yes, the headline score is optimistic and you need a held-out set.
• Check who signs off. A generated finding reaching a client without a named human reviewer is a professional-standards problem, not a technical one.