async, def and the event loop: how one line freezes the whole API

Agents love async def. It appears at the top of nearly every endpoint they write, whether or not the body contains a single await. Most of the time it makes no difference. Occasionally it turns one slow call into a total outage: the dashboard hangs, the health check times out, Azure restarts the container, and there is no exception anywhere in the logs. This is the one FastAPI concept where a line that looks completely ordinary can take the whole service down, so it is worth understanding properly.

One thread, no interruptions

Inside each uvicorn worker process there is a single event loop: one thread holding a queue of work that is ready to run. It picks a coroutine, runs it, and keeps running it until that coroutine hits an await on something unfinished. Then it parks that coroutine and runs the next one.

Two consequences follow, and everything else in this topic comes from them:

1. await is the only handover point. Concurrency comes entirely from coroutines voluntarily handing the loop back. Code between two awaits has exclusive use of the thread.
2. There is no pre-emption. Nothing takes the loop away from running code. A synchronous call that takes two seconds stops every request in that process for two seconds — not only its own.

So async def by itself buys nothing. A function full of blocking calls and no awaits is simply blocking code that FastAPI has been told not to worry about.

What def versus async def means to FastAPI

FastAPI treats the two differently, and the difference is protective. The docs state it plainly: when you declare a path operation with plain def instead of async def, “it is run in an external threadpool that is then awaited, instead of being called directly (as it would block the server)”. The same applies to dependencies.

That gives a simple rule:

The dangerous quadrant is the one agents write by default: blocking calls inside async def.

The blocking calls to grep for

# BEFORE — every one of these lines freezes the whole worker
@router.get("/waves/{wave_id}/enrich")
async def enrich(wave_id: int):
    meta = requests.get(f"{PANEL_API}/waves/{wave_id}").json()  # blocking HTTP
    df = pandas.read_sql(QUERY, engine)                          # sync DB + CPU
    time.sleep(1)                                                # rate-limit "fix"
    body = open(f"/mnt/exports/{wave_id}.csv").read()            # blocking disk
    return score(df, meta, body)

The usual suspects: requests.*, time.sleep, synchronous database drivers (psycopg2, pyodbc, sync SQLAlchemy Session), pandas.read_sql / read_excel, plain open().read() on anything large, and synchronous vendor SDKs — including openai.OpenAI() rather than AsyncOpenAI().

The quickest correct fix is often to delete one keyword:

# AFTER (option 1) — plain def: FastAPI runs it in the threadpool
@router.get("/waves/{wave_id}/enrich")
def enrich(wave_id: int):
    meta = requests.get(f"{PANEL_API}/waves/{wave_id}").json()
    ...

# AFTER (option 2) — stay async, use async libraries, thread what's left
@router.get("/waves/{wave_id}/enrich")
async def enrich(wave_id: int, client: HttpClientDep):
    r = await client.get(f"{PANEL_API}/waves/{wave_id}")
    r.raise_for_status()
    df = await asyncio.to_thread(pandas.read_sql, QUERY, engine)  # off the loop
    return score(df, r.json())

asyncio.to_thread(fn, *args) runs fn in a separate thread and returns a coroutine you await. run_in_threadpool from fastapi.concurrency (re-exported from Starlette) does the same job using the same pool FastAPI uses for sync endpoints. Either is correct.

One caveat the Python docs are explicit about: because of the GIL (global interpreter lock, which allows only one thread to execute Python bytecode at a time), to_thread “can typically only be used to make IO-bound functions non-blocking”. Waiting on a network or a disk, yes. A pandas groupby over a million rows, no — that needs a separate process or a queue.

Doing HTTP properly

# BEFORE — new client per request: no connection pooling, no TLS reuse
async def fetch(url: str):
    async with httpx.AsyncClient() as client:
        return (await client.get(url)).json()

The HTTPX docs warn against exactly this: to benefit from connection pooling, do not instantiate a client per call. Create one at startup and inject it:

# AFTER — one client for the app's lifetime
@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.http = httpx.AsyncClient(timeout=30.0)
    yield
    await app.state.http.aclose()

app = FastAPI(lifespan=lifespan)

def get_http(request: Request) -> httpx.AsyncClient:
    return request.app.state.http

HttpClientDep = Annotated[httpx.AsyncClient, Depends(get_http)]

Always set a timeout. A request with no timeout inside a pipeline is how a single unresponsive upstream turns into a stuck job that never finishes.

Real concurrency: gather, with a limit

Once you are genuinely async, asyncio.gather is the payoff. Forty independent calls take roughly as long as the slowest, not the sum.

# BEFORE — 3,000 concurrent calls at once: 429s, socket exhaustion, memory spikes
summaries = await asyncio.gather(*[summarise(d) for d in docs])

# AFTER — bounded fan-out, failures kept rather than lost
sem = asyncio.Semaphore(10)          # at most 10 in flight

async def summarise_bounded(doc):
    async with sem:
        return await summarise(doc)

results = await asyncio.gather(*(summarise_bounded(d) for d in docs),
                               return_exceptions=True)
ok = [r for r in results if not isinstance(r, Exception)]
failed = [(d.id, r) for d, r in zip(docs, results) if isinstance(r, Exception)]

Two things to know about gather. With the default return_exceptions=False, the first exception propagates immediately but the other awaitables are not cancelled and keep running in the background. With return_exceptions=True, exceptions come back as ordinary items in the results list, in input order — which means the very next line must check isinstance(r, Exception). Agents add the flag to stop a crash and then forget the check, so exception objects get iterated as if they were data.

Spotting a blocked loop in production

A blocked loop has a distinctive signature, and it looks nothing like a normal error:

• Everything slows down together, including endpoints that touch nothing. A /health route that returns a literal starts taking seconds.
• CPU stays low. The process is waiting, not working. High latency with 10% CPU is the giveaway.
• No exceptions. Nothing raised; requests merely queued.
• Then the platform health probe fails, Azure restarts the container, and you get 502s and a restart loop — with the real cause nowhere in the logs.

// Application Insights: does a trivial endpoint slow down with the rest?
requests
| where timestamp > ago(3h)
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95),
            count() by name, bin(timestamp, 5m)
| where name has "health" or name has "enrich"
| order by timestamp asc

If the health route’s p50 climbs at the same moment as the heavy route’s, the loop is blocked; if only the heavy route climbs, you have an ordinary slow endpoint.

The SQLAlchemy trap

One combination deserves a specific warning. If the repo uses create_engine and Session — the synchronous API — then endpoints touching the database should be plain def. An agent “optimising” by adding async to every endpoint moves every query onto the loop, and the API’s throughput collapses to one database round trip at a time across the whole worker. The async path is create_async_engine, AsyncSession and await session.execute(...), and it needs eager loading (selectinload) for relationships or you will meet MissingGreenlet. What you must never have is one style calling the other inside a single request.

What to check in your own code

• Grep for requests. and time.sleep in files that also contain async def.
• Grep for async def endpoints with no await in the body — either the async is pointless or something blocking is hiding.
• Grep for create_engine/Session alongside async def on the same endpoint.
• Grep for httpx.AsyncClient( inside a function rather than in the lifespan handler, and for any HTTP call with no timeout.
• Grep for asyncio.gather and check each one for a Semaphore (or batching) and for an isinstance(..., Exception) check after return_exceptions=True.
• Grep for openai.OpenAI( inside async def — it should be AsyncOpenAI there.
• Chart p95 latency for your /health endpoint next to your heaviest endpoint. If they move together, you have found your outage.