Debugging without a debugger

An agent built you a pipeline that pulls survey responses, runs an LLM step over each open-ended answer and writes the results to a table behind a dashboard. It has worked for three weeks. This morning the dashboard shows 412 responses where yesterday it showed 3,000, and nothing in the logs says “error”. You did not write this code, you cannot hold all of it in your head, and there is no obvious place to start.

There is a method. It is not clever, and it works on code you have never read.

The method in one shape

Debugging is a search problem. Somewhere between “input arrives” and “output is wrong” there is a first point at which reality diverges from what you expect. Every technique below exists to find that point faster. The loop is:

1. Reproduce it on demand.
2. Read what the system already told you, properly.
3. Form one hypothesis, stated as a prediction.
4. Narrow — halve the space the bug can be in.
5. Repeat 3 and 4 until you can say “by here it is right, by there it is wrong”.
6. Only then fix it, and confirm the reproduction now passes.

The two things people skip are steps 1 and 3. Skipping reproduction means you can never tell a fix from a coincidence. Skipping the single hypothesis means you change five things at once and learn nothing from the result.

Reproduce it, in seconds if possible

You want a command or a click sequence that makes the bug happen every time. Short is better than realistic: a failing test beats a manual run through the UI, because you will do it forty times.

If it happens only sometimes, that is not a different kind of bug, it is a reproduction you have not found yet. The job becomes: what is different about the failing cases? One particular respondent, one payload shape, one time of day, one batch size. Intermittent is a clue, not a dead end.

# Turn the bug into a test. This is the single highest-leverage debugging move.
def test_long_open_ended_answer_is_not_dropped():
    row = {"id": "R_812", "answer": "x" * 32_000, "wave": 3}
    result = normalise_response(row)
    assert result["answer"]            # currently empty — this is the bug

Once it is a test, the whole rest of the method unlocks: you can bisect with it, you can refactor safely, and you will know the day it regresses.

Read the error properly

Read a Python traceback from the bottom. The last lines are the facts: the exception type, the message, and the innermost frame. Above that is the chain of calls that got there, and you read upwards until you hit a file you own.

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
  ...
  File "/app/app/routers/responses.py", line 58, in list_responses     <-- your code
    rows = [normalise_response(r) for r in raw]
  File "/app/app/services/normalise.py", line 22, in normalise_response  <-- your code, deepest
    return {"answer": row["answer"].strip(), "wave": row["wave"]}
KeyError: 'answer'                                                      <-- the fact

The useful line is normalise.py:22, and the useful question is “which row has no answer key?” — not “why is uvicorn broken”. Framework frames appear in every traceback and carry no information.

Two habits that make tracebacks far more useful:

# Agent's version — throws away the traceback and the original cause
try:
    row = normalise_response(raw)
except Exception as exc:
    logger.error(f"normalise failed: {exc}")
    raise HTTPException(status_code=500, detail="error")

# Better — keep the traceback, keep the cause, record which row
try:
    row = normalise_response(raw)
except Exception:
    logger.exception("normalise failed", extra={"row_id": raw.get("id")})  # logs full traceback
    raise HTTPException(status_code=500, detail="Could not process response") from None

logger.exception records the traceback; logger.error(str(exc)) records one line of it. And raise ... from exc preserves the chain so you see both the original error and the wrapper.

The worst thing in agent-written pipelines is the silent swallow:

for row in rows:
    try:
        write(normalise_response(row))
    except Exception:
        pass          # <-- this is your missing 2,588 responses

That is the code that produces “412 where there should be 3,000, and no errors”. Grep for except Exception: pass and except: pass across every service you run, right now.

Narrow: bisect by commit, or by comment

If it used to work, history knows where it broke. git bisect binary-searches your commits: about six tests to search sixty commits.

git bisect start
git bisect bad                      # the current, broken commit
git bisect good v1.4.0              # a commit you know was fine
# git checks out the midpoint; you test; then:
git bisect bad      # or: git bisect good
# ...until git names the first bad commit
git bisect reset                    # always finish with this

With a pass/fail command it runs itself, unattended:

# exit 0 = good, exit non-zero = bad
git bisect start HEAD v1.4.0
git bisect run pytest tests/test_normalise.py -q

When there is no useful history — the whole pipeline arrived in one agent commit, or it never worked — bisect by comment instead. Cut the code in half inside the current version: stub a function to return a fixed value, comment out the LLM step, replace the network call with a hard-coded response, skip the write. Each cut tells you which half of the pipeline holds the bug.

def enrich_with_llm(text: str) -> str:
    return "STUB"     # temporary: does the bug survive without the LLM step?
    # ... real implementation below

This feels crude and is extremely effective, because it converts reasoning into an experiment.

Print and log deliberately

Print debugging is not a lesser technique; careless print debugging is. A useful print answers a question you actually asked, and identifies itself.

# Useless — which line printed this? what type is it? how many are there?
print(rows)

# Useful — Python's = specifier prints the expression alongside its value
print(f"AAA {len(rows)=} {type(rows[0])=} {rows[0].keys()=}")

Use a distinctive marker like AAA so you can grep the output and, more importantly, grep the source for leftovers before you commit.

In anything deployed, use the logger rather than print. Log calls carry a level, a timestamp and a module; they can be filtered, left in permanently, and turned up without a deploy. Prints from concurrent requests interleave into nonsense, and structured log pipelines often drop stdout entirely.

logger.info(
    "normalise batch complete",
    extra={"wave": wave, "received": len(raw), "written": len(rows), "dropped": len(raw) - len(rows)},
)

That one line would have caught the missing responses on day one. The pattern to copy: log counts in and counts out at every stage, so any stage that loses records announces itself.

Check the boundary

Most bugs in a FastAPI-plus-React system are not in either codebase; they are in what crosses between them. Rather than reasoning about it, look at the bytes.

-- Is the data missing, or is the query hiding it? Look at the row.
SELECT id, wave, length(answer) AS answer_len, created_at
FROM responses
WHERE wave = 3
ORDER BY created_at DESC
LIMIT 20;

A 401 in the Network tab, an empty [] response, a CORS error, a row that exists with a null column — each of these ends the guessing in one look and tells you which half of the system to stop reading.

Actual breakpoints, when you want them

You have real debuggers available, and they cost nothing to use.

In Python, breakpoint() on any line drops you into pdb with the live scope. Guard it with a condition to get the one iteration you care about:

for row in rows:
    if row["id"] == "R_812":
        breakpoint()      # stops once, with row, rows and locals all inspectable
    write(normalise_response(row))

At the prompt: p expr prints, pp pretty-prints, l lists nearby source, w shows the stack, u and d move between frames, n steps over, s steps into, c continues, q quits. Setting PYTHONBREAKPOINT=0 in your production environment neutralises every breakpoint() call in the codebase, which is a cheap safety net.

In the browser, the Sources panel gives you two things better than console.log: a conditional breakpoint (right-click a line number, add id === "R_812") and a logpoint, which logs an expression every time the line runs without pausing and without editing your source. Also switch on “Pause on uncaught exceptions” — it stops at the line that actually threw, which for a React error is far more informative than the component stack in the overlay. A fetch breakpoint on a URL substring pauses whenever that request is made, so you can see the call site.

When it only breaks in production

Note what the usual causes have in common:

1. Configuration — a missing secret, a URL pointing at staging, a feature flag set differently.
2. Data — real answers contain nulls, duplicates, emoji, right-to-left text and 32,000 characters. Your fixture had three tidy rows.
3. Scale and concurrency — timeouts, an exhausted database connection pool, two scheduled jobs overlapping, a race that needs two simultaneous requests.
4. Build differences — a minified bundle, production-mode React, a different Python or Node version in the image.
5. Network — egress rules, TLS, CORS, a private endpoint.

Four of the five are not in the code, which is why re-reading the code does not help.

So make the system talk. Query your logs for the failing requests and look for what they share:

These queries use the legacy table names, which work when you query the Application Insights resource itself. From a Log Analytics workspace the same data is AppRequests and AppExceptions, with OperationId and DurationMs instead of operation_Id and duration.

// Application Insights Logs: what do the failures have in common?
requests
| where timestamp > ago(24h) and success == false
| summarize failures = count() by name, resultCode, bin(timestamp, 1h)
| order by failures desc

exceptions
| where timestamp > ago(24h)
| project timestamp, type, outerMessage, operation_Id, customDimensions
| order by timestamp desc
| take 50

operation_Id is the correlation id: it lets you pull every log line, dependency call and exception for one request. Adding a correlation id to your own structured logs, and logging it on the front end too, is the single best investment for production debugging.

Then convert production into local. Take one real failing input — the payload from the Network tab, the exact row from the database, the JSON from the log’s custom dimensions — and paste it into a test. At that point you have a reproduction, and you are back at step 1 of the method with all the ordinary tools available.

What to check in your own code

• Grep every service for except Exception: pass, except: pass and except Exception: continue. Each one is a place where failures are invisible.
• Grep for logger.error(f"...{e}") and upgrade to logger.exception(...) so tracebacks survive.
• Check that each pipeline stage logs a count in and a count out; a stage that silently drops records is the commonest agent-written bug.
• Grep for leftover print( and console.log( in anything deployed.
• Confirm PYTHONBREAKPOINT=0 is set in your production app settings, and that no breakpoint() calls are committed.
• Pick one recent production error and try to trace it end to end by operation_Id in Application Insights. If you cannot, your logging needs a correlation id before your next incident, not during it.
• For each service, write down the one command that reproduces your last bug. If you cannot, you did not finish debugging it — you stopped when the symptom went away.