Background work and queues

An agent built you an endpoint that takes a batch of uploaded transcripts, runs an LLM coding pass over each one, and writes the results to your database. It returns instantly, which felt like magic. Then you deployed on a Wednesday afternoon and three research managers reported that their batches “finished” but produced nothing. Nothing in the logs said it failed, because from the API’s point of view nothing did fail. This article is about where long work should actually live, and what the record of it needs to look like.

Why long work has to leave the request

An HTTP request is a conversation with a deadline. The browser has a timeout. So does Azure’s load balancer, your reverse proxy, and the fetch call in your React app. A request that takes four minutes will be cut somewhere along that chain, and the user gets an error even though the work may have completed.

Worse, while a request is open it occupies a slot. FastAPI running under Uvicorn has a finite number of worker processes and, inside each, a finite thread pool. Ten concurrent four-minute jobs can make every other endpoint — including your health check — stop responding.

So the endpoint’s job is to accept the work and hand back a receipt. Something else does the work.

What BackgroundTasks really is

FastAPI’s BackgroundTasks runs a function after the response has been sent, in the same process:

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

@app.post("/notify/{email}", status_code=202)
async def notify(email: str, background_tasks: BackgroundTasks):
    # Runs after the response is delivered, in THIS process.
    background_tasks.add_task(write_audit_row, email, event="notified")
    return {"queued": True}

That is the whole feature. There is no storage, no worker, no retry, no visibility. An async def task runs on the same event loop as your requests; a plain def task is handed to Starlette’s thread pool. FastAPI’s own documentation says that for heavy computation you want “other bigger tools like Celery”, backed by a message queue.

The failure mode is specific: the task is a Python object in memory. A deploy, a container recycle, an App Service restart, a crash, or an out-of-memory kill takes the process — and every pending task with it. And because you already returned a success response, nobody finds out.

Here is the shape an agent commonly emits:

# BEFORE — four minutes of LLM work as an in-process background task
@app.post("/batches/{batch_id}/code")
async def code_batch(batch_id: int, background_tasks: BackgroundTasks):
    background_tasks.add_task(run_llm_coding, batch_id)  # 4 min, no record
    return {"status": "processing"}                      # a lie after a restart

And the shape you want:

# AFTER — record the job, enqueue it, return a receipt the front end can poll
@app.post("/batches/{batch_id}/code", status_code=202)
async def code_batch(batch_id: int, session: SessionDep, queue: QueueDep) -> JobOut:
    job = Job(kind="llm_coding", params={"batch_id": batch_id}, status="queued")
    session.add(job)
    await session.commit()          # the row exists before the message does
    await queue.enqueue("llm_coding", job_id=str(job.id))
    return JobOut.model_validate(job)

Note the order: commit the row, then enqueue. If you enqueue first and the commit fails, a worker picks up a job id that does not exist.

Workers and queues

A worker is a separate long-running process whose only job is to pull messages off a queue and run handlers. It is deployed alongside your API — a second Azure Container App, a second App Service, or a second container in the same image with a different start command — and it can be restarted, scaled and monitored independently.

A queue (or broker) is durable storage for messages. The Python options you will see:

Which one matters less than that there is one. On Azure the pull is often towards Queue Storage or Service Bus, because you already have the account and the managed identity, and there is no Redis to run.

# A worker handler, arq style — a plain async function plus a WorkerSettings class
async def llm_coding(ctx, job_id: str) -> None:
    async with session_factory() as session:
        job = await claim_job(session, job_id)   # queued -> running, or return None
        if job is None:
            return                               # already handled: duplicate delivery
        try:
            await run_coding(session, job)
            await finish_job(session, job, status="succeeded")
        except Exception as exc:                 # noqa: BLE001 — record, then re-raise
            await finish_job(session, job, status="failed", error=str(exc))
            raise                                # let the queue count the attempt

The job record

The queue is plumbing. Your job record is the thing users and dashboards read, and it belongs in your own database:

CREATE TABLE jobs (
  id            UUID PRIMARY KEY,
  kind          TEXT        NOT NULL,
  status        TEXT        NOT NULL,  -- queued|running|succeeded|failed|cancelled
  params        JSONB       NOT NULL,
  progress      JSONB,                 -- {"processed": 412, "total": 5000}
  result        JSONB,
  error         TEXT,
  attempts      INT         NOT NULL DEFAULT 0,
  idempotency_key TEXT      UNIQUE,    -- stops a double-click starting two runs
  heartbeat_at  TIMESTAMPTZ,           -- how you detect a dead worker
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  finished_at   TIMESTAMPTZ
);

Two columns there do heavy lifting. idempotency_key means a repeated submission returns the job that already exists. heartbeat_at is how you tell “still working” from “the worker died in July”.

The front end reads it through one endpoint:

@app.get("/jobs/{job_id}", response_model=JobOut)
async def get_job(job_id: UUID, session: SessionDep) -> Job:
    job = await session.get(Job, job_id)
    if job is None:
        raise HTTPException(status_code=404, detail="No such job")
    return job

And a React dashboard polls it, stopping itself when the job is done:

const TERMINAL = ["succeeded", "failed", "cancelled"];

const { data: job } = useQuery({
  queryKey: ["job", jobId],
  queryFn: () => fetch(`/api/jobs/${jobId}`).then((r) => r.json()),
  // Returning false switches polling off once the job reaches a terminal state.
  refetchInterval: (q) => (TERMINAL.includes(q.state.data?.status) ? false : 3000),
});

If the code you inherited uses setInterval inside a useEffect instead, check that the effect returns a cleanup that clears it. Without one, every re-render adds another timer and a tab left open overnight keeps polling a job that finished before lunch.

Idempotency, because delivery is at-least-once

Every queue worth using guarantees at-least-once delivery: if a worker stops acknowledging, the message comes back. It does not guarantee exactly once. So a handler that runs twice must leave the same end state.

# BEFORE — runs twice, charges twice, writes twice
await session.execute(insert(Export).values(survey_id=sid, rows=rows))
await session.execute(
    update(Survey).where(Survey.id == sid).values(export_count=Survey.export_count + 1)
)

# AFTER — keyed on the job, so the second delivery is a no-op
stmt = insert(Export).values(job_id=job.id, survey_id=sid, rows=rows)
await session.execute(stmt.on_conflict_do_nothing(index_elements=["job_id"]))
# Deterministic blob name: a re-run overwrites the same object instead of adding one
await container.upload_blob(f"exports/{job.id}.xlsx", data, overwrite=True)

The same idea applies to the status transition itself. Claim a job with a conditional update rather than a read-then-write, so two workers cannot both believe they own it:

UPDATE jobs SET status = 'running', attempts = attempts + 1, heartbeat_at = now()
WHERE id = :job_id AND status IN ('queued', 'running')
RETURNING id;   -- no row returned means someone else already claimed it

What production actually does to you

The visibility timeout. When a worker reads a message the queue hides it rather than deleting it. Finish, and the worker deletes it. Die, and the timeout expires and the message reappears. Set the timeout shorter than your slowest job and the queue will start a second copy of the job while the first is still running — duplicate LLM spend, interleaved writes, a progress bar going backwards.

Poison messages. One malformed transcript fails on every attempt and loops forever. Queues count deliveries; Service Bus dead-letters automatically after a maximum delivery count, while with Azure Queue Storage you check the dequeue count in your handler and park the message yourself. Either way you want a place where bad messages stop and stay visible.

Stuck rows. A worker killed mid-job leaves status = 'running' for ever, because nothing was alive to write anything else. A reaper — a scheduled query for jobs whose heartbeat_at is older than a few minutes — is the only thing that can correct it from outside.

Progress writes. Updating progress inside a 50,000-iteration loop turns a data job into a database-write job. Write every few hundred items, or every few seconds.

What to check in your own code

• Grep for add_task(. For each hit, ask how long the function runs and whether losing it silently would matter. Anything over a second or two, or anything that costs money, wants a queue.
• Check whether any endpoint that starts long work returns 202 and a job id. If it returns {"status": "processing"} with no id, the front end has no way to ever learn the truth.
• Confirm there is a jobs table (or equivalent) and that it has a terminal failed state that something actually writes. An agent will often implement the happy path only.
• Query for rows stuck in running older than your longest plausible job. If any exist, there is no reaper.
• Look at the order of operations in the endpoint: row committed before the message is enqueued, not after.
• Read one worker handler and ask what a second delivery of the same message would do. Look for plain INSERT, + 1 updates, unconditional emails and timestamped blob names.
• Check the queue’s visibility or lock duration against your slowest job, and find out where dead-lettered messages go and who looks at them.
• In the front end, search for setInterval and confirm each one is cleared in an effect cleanup, and that polling stops on a terminal status.