The Pipeline That Never Fails: Automation as a Claim About the World

Every automated pipeline you run is an argument. It says: if you take this data, apply these steps in this order, and read off the number at the end, that number is true enough to act on. Nobody writes that sentence down, which is a shame, because it is the only part that matters. The code is merely how the argument gets executed. When a scheduled job pulls survey responses at 6am, classifies open-ended answers, aggregates them by segment and pushes a figure into a dashboard, the pipeline is making a claim about what respondents said. A client will repeat that claim in a meeting. The question worth asking about any pipeline, then, is not “does it run?” but “when it is wrong, how would anyone find out?”

The failure you want is the loud one

There is a comforting fiction in automation: that a pipeline which has not alerted anyone is a pipeline that is working. The opposite is closer to the truth. A job that crashes has done you a favour. It has stopped, refused to produce output, and made its own failure your problem. You lose a morning. You do not lose credibility.

The dangerous failure is the silent wrong answer — a run that completes, reports success, writes plausible numbers, and is wrong. Consider the shapes this takes in research automation:

• The source API renames a field. What used to hold a weight now returns nothing, the code treats missing as zero, and every weighted average shifts slightly.
• A date filter means “the last 30 days” but the timezone is wrong by an hour, so each run quietly drops or double-counts the responses landing near midnight.
• A classification step used to return one of five categories. After a model change it occasionally returns a sixth, unmapped label, which falls into an “other” bucket nobody reads.
• A retry meant to be safe runs the aggregation twice, and a count column is double what it should be — but only on days when the network hiccuped.

None of these raise an exception. All of them produce a number. The pattern is the same each time: the code’s assumptions and the world’s behaviour drifted apart, and nothing was standing between them. Which is why a long unbroken green streak should provoke mild suspicion rather than pride. It might mean everything is healthy. It might mean nothing is checking.

Idempotency is the foundation of trust

If there is one property worth insisting on, it is this one. A step is idempotent when running it twice produces the same result as running it once. Not “runs twice without crashing” — produces the same state.

This matters because reruns are the normal condition of automation, not the exception. The network fails, the API rate-limits you, the model times out, someone notices yesterday’s data was incomplete and wants the day reprocessed. If you cannot rerun a step safely, then every failure becomes an investigation: what did it manage to do before it died, and what do I have to undo by hand? That investigation is where the real cost of a broken pipeline lives.

Here is the version an agent commonly writes:

# Runs fine. Run it twice and you have two copies of every row.
async def store_responses(rows: list[dict], conn) -> None:
    for row in rows:
        await conn.execute(
            "INSERT INTO responses (respondent_id, answer) VALUES ($1, $2)",
            row["respondent_id"], row["answer"],
        )

And the version that survives a rerun:

# A unique key on respondent_id turns "insert" into "make sure this exists".
async def store_responses(rows: list[dict], conn) -> None:
    for row in rows:
        await conn.execute(
            """
            INSERT INTO responses (respondent_id, answer)
            VALUES ($1, $2)
            ON CONFLICT (respondent_id) DO UPDATE SET answer = EXCLUDED.answer
            """,
            row["respondent_id"], row["answer"],
        )

The difference is a few words, and it changes what the pipeline is. The first version’s meaning depends on how many times it has run. The second version’s does not. The same instinct applies to aggregates: key each write to the day and segment it describes, so a repeat replaces rather than stacks.

A pipeline whose steps are all idempotent has a pleasant property — the correct response to almost any failure is “run it again”. A pipeline whose steps are not has no general recovery strategy at all, only a growing folder of one-off repair scripts.

Where a model step belongs, and where it does not

Language models made a previously impossible pipeline step easy: reading unstructured text and returning something structured. Coding 4,000 open-ended survey answers into themes; pulling entities out of a policy document; summarising a transcript. Those are real wins.

But a model step has a property no other step in your pipeline has: it is non-deterministic and confidently wrong. Give it the same input twice and you may get two answers. Give it input you did not anticipate and it will still return something well-formed. Compare that with a division or a date filter, which fails visibly when it fails at all.

So: a model step fits where the task is genuinely linguistic, the output can be validated against a closed set of options, and a wrong answer on one row is survivable. It does not fit where determinism matters, where arithmetic is involved, or where one wrong output corrupts everything downstream.

Notice the shape: models are good at proposing and bad at deciding. Where you do use one, constrain the output and validate it in code rather than trusting the prose.

from pydantic import BaseModel, field_validator

THEMES = {"price", "quality", "service", "delivery", "other"}

class Coding(BaseModel):
    theme: str
    confidence: float

    @field_validator("theme")
    @classmethod
    def known_theme(cls, v: str) -> str:
        v = v.strip().lower()
        if v not in THEMES:            # a new label is a pipeline bug, not an "other"
            raise ValueError(f"unexpected theme: {v!r}")
        return v

coding = Coding.model_validate_json(model_output)   # raises loudly on drift

The important line is the one that raises. An unmapped label silently swept into “other” is exactly the silent wrong answer described earlier. Raising converts it into the loud failure you can actually act on.

How would you know it got worse?

This is the question that separates a pipeline you can defend from one you merely operate. Suppose the classification step became noticeably less accurate next Tuesday — the provider updated a model, or your prompt was edited, or the incoming data shifted in character. What in your system would notice?

For most pipelines, honestly, the answer is: a client, eventually, in a meeting. That is a bad place to find out.

The fix is not glamorous. It is a gold set: a few dozen to a few hundred rows where you have decided, by hand, what the right answer is. You run those rows through the same step on every deployment and every model change, and you compare. It is a test suite for judgement rather than for logic.

# Cheap, boring, and the only thing that tells you the step got worse.
def evaluate(gold: list[tuple[str, str]], classify) -> float:
    correct = sum(1 for text, expected in gold if classify(text) == expected)
    return correct / len(gold)

score = evaluate(GOLD_SET, classify_verbatim)
assert score >= 0.88, f"classifier regressed: {score:.2%}"   # fail the deploy

Two things make this work. The gold set must be built from real, messy inputs, including the awkward ones — the sarcastic answers, the blank-but-not-empty ones, the ones in the wrong language. And the threshold has to be a number someone chose deliberately and is willing to defend, not whatever the first run happened to produce.

Alongside a gold set, cheap statistical tripwires catch a great deal for very little effort: row counts within an expected range, the share of answers per category, the proportion of nulls in key columns, the mean of a numeric field. You do not need clever anomaly detection. A rule saying “if today’s row count is under half the trailing average, stop and tell me” would have caught a surprising number of quiet disasters.

Being able to explain a number six months later

Here is the scenario that should shape your design more than any other. It is March. A client emails about a figure in a deck you sent in September. Where did 62 per cent come from?

To answer that, you need provenance: for any output, the ability to say which inputs produced it, which version of the code processed them, and when. Without it you have two options, both bad — rerun today’s pipeline and hope it still gives 62 per cent, or tell the client you cannot say.

Provenance does not require a platform. It requires that every run writes down a little about itself:

# Written once per run; the cheapest insurance in the whole pipeline.
run = {
    "run_id": run_id,                    # uuid, on every row this run writes
    "started_at": started_at.isoformat(),
    "code_version": os.environ["GIT_SHA"],   # which code did this
    "model": "the exact model id used, not 'latest'",
    "prompt_version": "verbatim-coding-v4",
    "source_window": [window_start.isoformat(), window_end.isoformat()],
    "rows_in": len(rows),
    "rows_out": len(results),
}

Then stamp run_id onto the rows the run produces. That single column turns an unanswerable question into a query, and makes the interesting questions tractable too: which runs used prompt v3, how the category mix changed when the model was swapped, which outputs came from the day the source API misbehaved.

Two related habits pay off. Pin your model ids rather than pointing at a moving alias — otherwise the meaning of your pipeline changes without a line of your code changing, which is a nasty thing to debug. And keep the raw input, unmodified, alongside the processed output. Storage is cheap. Reconstructing what an API returned in September is not possible at any price.

The checkpoints worth keeping

The temptation with a working pipeline is to remove the humans, because the humans are the slow part. Some should indeed go. A person clicking “run” on a job that runs the same way every day is not a checkpoint; it is a single point of failure that needs annual leave.

The ones worth keeping sit where a mistake becomes expensive or hard to reverse, and where human judgement genuinely beats a rule.

• Before anything leaves the building. Automated ingest, automated processing, automated draft — then a person looks before a client does. Almost always worth its cost.
• Before an irreversible write. Deletions, overwrites of historical data, anything that sends email. Automation that can destroy the record of its own mistake is a poor bargain.
• On anything the tripwires flagged. The value of a breach is not that it stops the pipeline; it is that it routes a specific, described anomaly to someone who can say “that is fine, it was a bank holiday” — or that it is not.
• On a sample, always. Reading ten real outputs a week is the only habit that keeps you honest about what your pipeline actually produces. Dashboards show aggregates, and aggregates hide exactly the failures described at the top of this piece.

What makes these different from bureaucratic sign-off is that each has an answer to what would I do differently as a result of looking? If there is no answer, the checkpoint is theatre and should go.

What to check in your own pipelines

Concrete things to look for in code an agent wrote for you:

• Find every INSERT and ask what happens on a rerun. No ON CONFLICT, no unique key, no delete-then-insert in a transaction means the step is not idempotent.
• Search for except Exception followed by pass, continue or a bare log line. Each is a place where a failure became a silent wrong answer.
• Look for or 0, or "" and .get("field", 0) on numeric fields from external sources. A missing value treated as zero is the classic quiet corruption.
• Check whether the model id is pinned or an alias like latest. Aliases mean behaviour can change while your repository does not.
• Ask of any output number: which run produced it, with which code, from which inputs? If that is not a query you could run, add a run_id.
• Confirm a gold set exists for every model-judgement step, and when it was last run. A gold set nobody runs is a comment.
• Check the schedule’s timezone handling around daylight saving. A “daily” job defined in local time silently skips or repeats an hour twice a year.

A pipeline is not infrastructure you maintain. It is a standing assertion you are making, on a schedule, to people who trust you. Idempotency, tripwires, a gold set and a run id are what let you keep making that assertion with a straight face.