Testing a FastAPI app

An agent added forty tests to your research API and they all pass, which feels reassuring until you notice that most of them assert a GET returns 200. Meanwhile the endpoint that lets any signed-in user fetch any project by id has no test at all. Tests are not a virtue you accumulate; they are a way of pinning down the specific decisions you do not want quietly changed. This article covers the machinery, and then which tests are worth having.

Calling your app without a server

pytest finds files named test_*.py, functions named test_*, and treats a failed assert as a failure. That is nearly the whole framework.

FastAPI’s TestClient calls your app in-process — no port, no network, no running Uvicorn:

from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_list_surveys():                 # plain def, not async def
    response = client.get("/api/surveys")
    assert response.status_code == 200
    assert response.json() == []

FastAPI’s documentation is explicit that test functions using TestClient should be normal def functions with no await, even when the endpoints themselves are async. The client runs your app in its own event loop and hands you an ordinary response.

That breaks down as soon as the test body needs to await something — an async session to seed a row, an async helper to assert against. Then you use httpx directly:

import pytest
from httpx import ASGITransport, AsyncClient
from app.main import app

@pytest.mark.anyio                       # FastAPI's docs use the anyio plugin
async def test_creates_job():
    async with AsyncClient(
        transport=ASGITransport(app=app), base_url="http://test"
    ) as ac:
        response = await ac.post("/api/jobs", json={"kind": "export"})
    assert response.status_code == 202

One trap worth memorising: AsyncClient with ASGITransport does not trigger your app’s lifespan. If your engine, HTTP client or cache connection is created in the lifespan handler, it will be None in these tests. Build those things in fixtures instead, or wrap the app with LifespanManager from asgi-lifespan. (TestClient used as a context manager does run lifespan.)

Fixtures

A fixture is a function that prepares something a test needs. A test asks for it by naming it as a parameter; a fixture that yields runs the rest of its body as teardown, pass or fail. Put shared ones in conftest.py and every test in that directory and below can use them without importing anything.

import pytest
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine

@pytest.fixture(scope="session")         # expensive, immutable: build it once
def anyio_backend():
    return "asyncio"

@pytest.fixture(scope="session")
def engine():
    return create_async_engine(TEST_DATABASE_URL)

Scope is where fixtures go wrong. function (the default) rebuilds per test; session builds once. Expensive and immutable belongs in session; anything a test mutates belongs in function. A session-scoped fixture holding mutable state is the standard explanation for tests that pass alone and fail together.

A test database that cleans up after itself

The pattern you want creates the schema once and wraps each test in a transaction it throws away:

@pytest.fixture
async def session(engine):
    async with engine.connect() as conn:
        trans = await conn.begin()                  # outer transaction
        factory = async_sessionmaker(bind=conn, expire_on_commit=False)
        async with factory() as s:
            yield s
        await trans.rollback()      # everything the test wrote disappears

No cleanup code to forget, no rows leaking into the next test, and it is fast because nothing is created or dropped. The subtlety: if your application code calls commit(), that can end the outer transaction. SQLAlchemy’s documented “joining a session into an external transaction” recipe handles this with a nested transaction (a SAVEPOINT) that is restarted after each commit — worth reading before you conclude rollback does not work for you.

Use the same database engine as production. An in-memory SQLite substitute for a PostgreSQL app hides exactly the differences — JSON types, case sensitivity, constraint behaviour, ON CONFLICT support — that break you later.

Dependency overrides

app.dependency_overrides is a dictionary mapping a dependency callable to a replacement. It is the supported seam for swapping the database session, the current user, or an outbound client:

from app.deps import current_user, get_session

@pytest.fixture
def client(session):
    async def _session_override():
        yield session                      # the rolled-back test session

    app.dependency_overrides[get_session] = _session_override
    app.dependency_overrides[current_user] = lambda: User(id=1, roles=["analyst"])
    with TestClient(app) as c:             # context manager -> lifespan runs
        yield c
    app.dependency_overrides.clear()       # or every later test runs as this user

That last line matters more than it looks. dependency_overrides is mutable state on the app object and persists for the life of the process. A leaked admin override makes your authorisation tests pass for the wrong reason — the worst kind of failure, because it is invisible.

Prefer overrides to monkeypatch for anything that is a dependency. Monkeypatching a name that was imported with from module import thing often does nothing, because the importing module already holds its own reference. Keep monkeypatch for environment variables, clocks and module-level constants.

Mocking outbound HTTP and LLM calls

No unit test should leave the machine. Real calls are slow, cost money, need live credentials, and fail when somebody else deploys. Intercept at the transport layer:

import httpx, respx

@respx.mock
def test_summariser_handles_truncated_json(client):
    respx.post("https://my-openai.openai.azure.com/openai/v1/responses").mock(
        return_value=httpx.Response(200, json={"output_text": '{"theme": "pri'}),
    )
    r = client.post("/api/transcripts/7/summarise")
    # The point of the test: bad model output must fail loudly, not write a half row
    assert r.status_code == 502
    assert "could not parse" in r.json()["detail"].lower()

The alternative — often cleaner — is to hide the call behind a small class or protocol, inject it as a dependency, and override it with a fake in tests. Either way, write three cases per outbound call: a good response, an error status, and a timeout.

Which raises the question of what is even testable about a non-deterministic model. Not the prose. Test everything around it:

• the prompt is assembled from the right fields (assert on the captured request body);
• a valid response parses into your Pydantic model via model_validate;
• malformed, truncated or empty output raises a clear error instead of writing nulls;
• retries are bounded and do not loop forever;
• a refusal is handled as a refusal, not as a summary.

Model quality is evaluated separately, with a graded sample set. That is a different activity from a test suite and should not be allowed to make CI red at random.

What is worth testing in agent-written code

Spend your effort where an agent guesses:

def test_cannot_read_another_tenants_project(client_as_tenant_b, project_in_tenant_a):
    r = client_as_tenant_b.get(f"/api/projects/{project_in_tenant_a.id}")
    assert r.status_code == 404          # not 200, and not a leaked 403 with a body

That test is four lines and covers the most common real data leak there is.

Reading a CI failure

A CI log is long and almost none of it is for you. Work backwards:

1. Jump to the end. pytest prints a short test summary info block listing the failed test ids. That is your list of problems, and often it is one problem reported five times.
2. Find that test’s section. Read the assertion line and the error, then scan the traceback for the last frame in your code — ignore the framework frames above and below it.
3. Reproduce it narrowly: pytest -x -q tests/test_jobs.py::test_claims_once. -x stops at the first failure, -q cuts the noise, --lf re-runs only last-failed.
4. If it passes locally, suspect environment or order. A KeyError on an environment variable means CI does not have a setting your code needs — which is real information, not a test problem. A failure that depends on which tests ran first means shared state; -p no:randomly and running the pair in the failing order will show you which.

pytest -x -q --lf                        # just the tests that failed last time
pytest tests/test_auth.py -k "tenant" -vv  # one file, matching names, full diffs
pytest --durations=10                    # find the tests making CI slow

If a test is genuinely flaky, fix or delete it. A suite people have learned to re-run is a suite that no longer tells anyone anything.

What to check in your own code

• Count the tests that assert only a status code. That number tells you how much of the suite is decoration.
• Grep the tests for httpx, requests, openai and your Azure endpoints without a respx or mock nearby: those tests are making real calls.
• Check that every fixture setting dependency_overrides clears it in teardown.
• Look for scope="session" or scope="module" on any fixture that returns something mutable, especially a session or a client holding auth state.
• Confirm the test database is the same engine as production, and that isolation comes from rollback rather than from DELETE statements that can fail halfway.
• Search for at least one test per endpoint that takes a resource id in the path, asserting a foreign id is refused.
• Check whether async tests use ASGITransport, and if so whether anything built in the app’s lifespan is also built in a fixture.
• Run pytest --durations=10 once. A single slow test usually means a real network call or a sleep.
• Look at the CI workflow file and confirm it fails the build on test failure rather than on a step that always exits zero.