LLMs in production: cost, latency and guardrails

A prototype that works in a notebook is a long way from a feature that thousands of people use every day. In production, an LLM call is a slow, metered, rate-limited dependency on someone else’s service, and its behaviour can shift under you. This article covers the operational craft: what drives cost and latency, how to cut both, how to keep bad inputs and outputs in check, and what to watch once it’s live.

Token economics

Most hosted models bill per token, with input and output tokens priced separately. Output tokens are typically several times more expensive per token than input tokens, and they’re also what makes responses slow. Prices change often, so reason in ratios and volumes rather than memorising numbers.

A rough mental model for a request’s cost:

cost = input_tokens x input_price + output_tokens x output_price

Things that quietly inflate it:

Long system prompts and few-shot examples repeated on every call.
Retrieving far more context than the answer needs.
Verbose outputs, or reasoning modes that generate many hidden tokens.
Multi-step agents that re-send the growing conversation at each step.
Retries, especially unbounded ones.

Estimate cost per request and per user action (one click may trigger several calls), then multiply by expected volume before launch.

Latency and streaming

Two numbers matter:

Time to first token. Affected by input length, queueing and model size.
Time per output token. Total time grows roughly linearly with response length.

Ways to go faster: use a smaller model where quality allows, shorten inputs, cap output length, run independent calls in parallel, and cache.

Streaming sends tokens as they’re generated. It makes the experience feel far faster because users start reading almost immediately, but the full response takes just as long and costs the same. It also complicates validation: if you display text as it arrives, output checks either run on partial text or after the user has already seen it. For structured data that code acts on, it’s often simpler to wait for the complete response.

Prompt caching

Providers can cache the processed prefix of a prompt. When a later request begins with exactly the same tokens, that part is served from cache: billed at a steep discount and processed faster. Some providers apply this automatically; others ask you to mark cache breakpoints. Caches expire after a period of inactivity and usually require a minimum prefix length, so check your provider’s documentation.

The design rule is the same everywhere: stable content first, variable content last.

Tool definitions and system instructions.
Shared reference documents or long examples.
Conversation history.
The new user message.

A timestamp or user name at the very top of the prompt will break the match for everything after it. Watch the cache-hit fields in API usage data to confirm it’s working.

Routing between models

Not every request needs your most capable model. Routing sends each request to the cheapest model likely to handle it:

Rules-based: simple classification or extraction goes to a small model; long, open-ended reasoning goes to a large one.
Classifier-based: a lightweight model or heuristic predicts difficulty.
Cascade: try the small model first and escalate if output fails validation or a quality check.

Routing lowers average cost and latency, but adds complexity and new failure modes. Evaluate each route on the traffic it actually receives.

Rate limits and resilience

Providers limit requests and tokens per minute. Plan for hitting them:

Retry rate-limit and overload errors with exponential backoff and jitter, never in a tight loop.
Queue non-urgent work, and use batch APIs where offered for offline jobs.
Set per-user or per-tenant quotas so one heavy user can’t starve everyone else. OWASP calls uncontrolled usage Unbounded Consumption.
Decide on a degraded mode: a fallback model or provider, a cached answer, or a clear “try again shortly” message.

Guardrails

Guardrails are checks around the model call.

Input guardrails screen what goes in: prompt injection patterns, off-topic or abusive requests, oversized inputs, and sensitive data that shouldn’t leave your system.

Output guardrails screen what comes out: schema validity, policy violations, leaked secrets or personal data, claims not supported by retrieved sources, and whether a requested action is permitted.

Guardrails reduce risk; they don’t eliminate it. Prompt injection in particular has no reliable filter, so limit what the model is allowed to do, not just what it’s allowed to say.

Every check costs latency and money and will sometimes block legitimate requests. Measure false positives as seriously as misses.

Monitoring and quality drift

Quality can change even when your code doesn’t. Providers update models behind aliases and retire old versions, users start asking different things, and retrieval sources go stale. Pin model versions where you can, and treat model upgrades like any other deployment: run your evals first.

Monitor at least:

Tokens and cost per request and per feature.
Latency percentiles, not just averages.
Error, refusal, validation-failure and fallback rates.
User signals: thumbs, edits, retries, abandonment.
Quality scores on a sample of live traffic, via automated evals or human review.

Tag everything with model and prompt version so you can tie a change in metrics to a change in the system.

Privacy

Every call sends data to a third party. Before shipping, find out:

Whether the provider retains inputs and outputs, for how long, and whether they may be used for training under your plan.
Where the data is processed, and whether contracts and regulations allow it.
Whether you can minimise or redact personal data before sending it.

Don’t forget your own side. Prompt logs are extremely useful for debugging and evals, and are also a pile of sensitive user data. Apply the same retention limits and access controls you’d apply to any other personal data.