Retries, timeouts and backoff

An agent wrapped your flaky vendor call in @retry and the errors stopped appearing in the logs. Two months later the nightly job shows as “running” at 11am, has written nothing, and the next run never started. Separately, a step that POSTs to an internal API produced two rows for every timeout.

Both are the same story. Retries and timeouts are the two halves of one decision — how long to wait, and what to do when waiting doesn’t help — and agents routinely add the first without the second.

Which failures deserve a retry

A retry is only useful if the identical call could plausibly succeed, unchanged, in a few seconds. That splits errors cleanly.

The right-hand column is where retrying does real damage: it turns a clear, fast failure into a slow one, and it burns quota to arrive at the same answer. The version agents write catches everything:

from tenacity import retry

@retry  # tenacity's documented default: retry forever, no wait, on any exception
async def fetch_page(cursor: str) -> dict:
    return (await client.get("/responses", params={"cursor": cursor})).json()

That is not a retry policy, it is an infinite loop with extra steps. A ValidationError from a schema change will spin until the platform kills the worker — and since the exception never escapes, nothing is logged as a failure.

import logging
from tenacity import (retry, stop_after_attempt, wait_exponential_jitter,
                      retry_if_exception_type, before_sleep_log)
import httpx

logger = logging.getLogger(__name__)

@retry(
    stop=stop_after_attempt(4),                          # 4 attempts, then give up
    wait=wait_exponential_jitter(initial=1, max=30),     # jittered, capped backoff
    retry=retry_if_exception_type((httpx.TransportError, httpx.HTTPStatusError)),
    before_sleep=before_sleep_log(logger, logging.WARNING),  # every retry is visible
    reraise=True,                                        # raise the real error, not RetryError
)
async def fetch_page(cursor: str) -> dict: ...

Four decisions, all explicit: when to stop, how long to wait, what counts as retryable, and how you find out it happened. reraise=True matters more than it looks — without it your except blocks see tenacity’s RetryError and your error handling quietly stops matching.

Backoff, jitter and the thundering herd

Exponential backoff doubles the wait between attempts (1s, 2s, 4s, 8s), giving a struggling dependency progressively more room. Jitter randomises each wait.

Jitter is not a refinement. If a vendor API has a 30-second wobble, every one of your callers fails at the same instant, and without jitter they all retry at the same instant, then again together, in synchronised waves that keep the service down. Google’s SRE book puts the rule flatly: always use randomised exponential backoff, because a small perturbation like a network blip otherwise causes retry ripples that all land together.

Backoff must also be capped. Uncapped doubling reaches minutes, which means a call that would have succeeded on attempt five is still sleeping when your job’s deadline arrives.

# Fixed sleep: synchronised retries, and the delay never adapts.
for attempt in range(5):
    try:
        return await call()
    except httpx.TransportError:
        await asyncio.sleep(2)   # every client in the fleet wakes together

# tenacity's equivalents, if you prefer to see the maths:
#   wait_exponential_jitter(initial=1, max=30)   -> 2^n seconds, capped, plus jitter
#   wait_random_exponential(multiplier=1, max=60) -> uniform 0..2^n, capped

Retry amplification and retry budgets

The dangerous property of retries is that they multiply across layers, they do not add. Google’s SRE book gives the canonical example: if a JavaScript client, a front end and a backend each make 4 attempts, one user action can produce 4 x 4 x 4 = 64 attempts at the database. Sixty-four, not twelve.

So retry in one place — the layer closest to the failure — and have the others propagate the error upward. If your FastAPI endpoint retries a database helper that itself retries, remove one of them.

Even one layer of retries needs a ceiling across the whole process, rather than per call. A per-call limit of three means that during a full outage you send three times your normal load at an already failing dependency. That is a retry budget: allow retries only while they stay under some share of total requests (a common figure is 10%), or under a fixed ceiling — the SRE book suggests something like 60 retries per minute per process — and fail immediately once it is spent.

import time

class RetryBudget:
    """Token bucket: retries are only allowed while tokens remain."""
    def __init__(self, per_minute: float = 60):
        self.rate, self.tokens, self.ts = per_minute / 60, per_minute, time.monotonic()

    def allow(self) -> bool:
        now = time.monotonic()
        self.tokens = min(60.0, self.tokens + (now - self.ts) * self.rate)
        self.ts = now
        if self.tokens < 1:
            return False        # budget spent: fail fast so the dependency can recover
        self.tokens -= 1
        return True

A circuit breaker is the next step up. A retry decides what to do about one failed call; a breaker decides whether to attempt calls at all. After a threshold of failures it opens and fails fast without touching the dependency, then after a cooldown lets one trial request through (half-open) and closes if it succeeds. It protects the dependency from you and protects your run from spending itself on timeouts.

Timeouts at every layer

Here is the failure that produced the 11am hang:

import requests   # smell 1: blocking client inside an async pipeline

def fetch_summary(doc: str) -> str:
    # smell 2: no timeout. requests waits forever by default.
    r = requests.post(LLM_URL, json={"input": doc})
    return r.json()["summary"]

With no deadline the socket read waits indefinitely. The worker never returns, so it never releases its database connection or its queue lease. The scheduler sees a job still “running” and won’t start the next one. One missing timeout turned one slow dependency into a stalled pipeline, and LLM calls are the slowest thing in most pipelines.

import httpx

# httpx defaults to 5s of network inactivity; be explicit anyway, and separate
# the connect budget from the read budget - LLM responses are legitimately slow.
timeout = httpx.Timeout(90.0, connect=5.0)

async with httpx.AsyncClient(timeout=timeout) as client:
    r = await client.post(LLM_URL, json={"input": doc})
    r.raise_for_status()

httpx.Timeout distinguishes four waits, and all four can hang you: connect (establishing the socket), read (waiting for the next chunk of response), write (sending the request body) and pool (waiting for a free connection from the pool). A pool timeout is the one people forget, and it is how a slow dependency starves requests that have nothing to do with it.

Every layer needs its own deadline:

-- Postgres: stop a runaway query rather than letting it hold a connection all night.
SET statement_timeout = '30s';

# SQLAlchemy 2.0: cap how long a worker waits for a pooled connection.
engine = create_async_engine(DSN, pool_size=5, max_overflow=5, pool_timeout=10)

Plus the layers you do not control: the gateway or App Service in front of your API, the queue’s visibility timeout, and the platform’s maximum job duration. The rule is that inner timeouts must be shorter than outer ones. If your HTTP client waits 120 seconds behind a gateway that gives up at 60, the gateway kills the request before your code can log a clean error or write a dead-letter row.

Pick the number from measurements

Choose a timeout from the dependency’s observed latency, not from a round number that feels generous. Take its high percentile under normal load — p99, or p99.9 for something you cannot afford to fail — and set the timeout somewhat above it. Set it too tight and you convert healthy-but-slow successes into failures and then into retries, which is how a latency wobble becomes an outage. Too loose, and you have no timeout at all in the cases that matter.

Then check the arithmetic against the enclosing deadline. Four attempts at a 90-second timeout, plus jittered backoff up to 30 seconds, is a worst case near seven minutes. Inside a five-minute job limit, that policy guarantees the worker is killed mid-write.

429 and Retry-After

429 Too Many Requests means you are being rate-limited (RFC 6585, section 4). It is worth retrying, but only after waiting — and if the response carries a Retry-After header, the server has told you exactly how long. RFC 9110 defines its value as either a number of seconds or an HTTP-date, so handle both:

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(resp: httpx.Response) -> float | None:
    raw = resp.headers.get("retry-after")
    if raw is None:
        return None
    if raw.strip().isdigit():
        return float(raw)                       # delay-seconds form
    when = parsedate_to_datetime(raw)           # HTTP-date form
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

Prefer that value over your own backoff curve: the server knows when its window resets and you are guessing. Retrying a 429 immediately is the standard way to get an API key throttled harder or blocked, and rotating keys to dodge a documented limit is abuse, not engineering.

Why retrying a non-idempotent step duplicates data

This is the part that surprises people. A timeout tells you the response never arrived. It does not tell you the work never happened.

@retry(stop=stop_after_attempt(3))
async def record_response(payload: dict) -> None:
    # If this returns 201 and then the connection drops, the row exists
    # and the retry creates a second one.
    r = await client.post("/responses", json=payload)
    r.raise_for_status()

The server may have committed the insert, sent the email or charged the card and then been slow to reply. Retrying repeats an effect that already took place. You cannot fix this by tuning the timeout or narrowing the exception list, and dropping retries only trades duplicates for lost data.

The fix is at the write. Either send a key the server deduplicates on:

key = natural_key(payload)           # stable: same payload, same key, every attempt
r = await client.post("/responses", json=payload, headers={"Idempotency-Key": key})

or, if you own the destination, make the write an upsert on a natural key so any number of attempts leaves one row. Retries are only safe on top of idempotence. Adding a retry decorator to a non-idempotent step is not hardening it — it is scheduling the duplicates.

What to check in your own code

• Grep for @retry with no arguments. Tenacity’s default retries every exception forever with no wait; that is unfinished code, not a policy.
• Grep for requests.get, requests.post, httpx.get and urlopen with no timeout=. Each is a place a run can hang indefinitely. (requests waits forever by default.)
• For each retried call, ask: is the effect idempotent? If it is a POST or a plain INSERT, the retry duplicates.
• Count the layers that retry the same logical call — browser, endpoint, helper, SDK. Attempts multiply; keep retries at one layer.
• Multiply attempts by the per-attempt timeout, add the backoff waits, and compare with the job’s or gateway’s deadline. If the total is larger, the worker gets killed mid-write.
• Check whether retryable exceptions are listed explicitly, or whether a bare except Exception is feeding the retry. A validation error should fail fast.
• Look for Retry-After handling anywhere you call a rate-limited API. If it is absent, you are guessing at a number the server already gave you.
• Check the database side: a statement_timeout and a pool pool_timeout are as important as the HTTP one, and are almost never set by agents.
• Confirm retries are logged. before_sleep=before_sleep_log(...) turns invisible flakiness into a number you can watch.