Retrieval-augmented generation (RAG)

A language model’s knowledge is a blurry snapshot of its training data. It does not know your internal wiki, last week’s price change, or the contract a customer signed. Retrieval-augmented generation (RAG) is the standard answer: before the model responds, search a collection of documents for relevant passages and put them in the prompt. The model then answers from text you supplied, not only from memory. The idea is simple; making it reliable is mostly a search problem.

Why ground a model in documents

The term comes from Lewis et al. (2020), who paired a text generator with a learned retriever over Wikipedia and found more specific, factual answers on knowledge-heavy tasks. In today’s applications RAG usually means a looser pattern — retrieve, then prompt — but the motivations are the same:

• Freshness. Update a document and the next answer reflects it. No retraining.
• Private knowledge. Models were never trained on your company’s data.
• Verifiability. Answers can cite the passages they came from.
• Scale. A corpus far larger than any context window can still be searched.

RAG is not a fix for everything. If you need a model to adopt a style or follow a format, prompting or fine-tuning fits better. RAG is for supplying facts.

The pipeline

1. Chunking

Documents are split into chunks small enough to retrieve precisely and fit several into a prompt. This is a real trade-off: large chunks dilute relevance and waste context; tiny chunks lose the surrounding meaning. Splitting on the document’s own structure — sections, paragraphs, functions in code — usually beats blind fixed-size splitting. Attaching the document title and section heading to each chunk, or overlapping neighbours slightly, preserves context that would otherwise be cut off.

2. Embedding and indexing

For semantic search, each chunk is passed through an embedding model that returns a vector; texts with similar meaning end up close together. Vectors go into an index built for fast approximate nearest-neighbour search. In parallel, many systems also build a classic keyword index.

3. Retrieval

At query time the user’s question (sometimes rewritten first) is embedded with the same model, and the nearest chunks are returned. Keyword retrieval runs the query against the inverted index.

4. Reranking

First-stage retrieval is built for speed, so it is coarse. A common upgrade is to retrieve a broad set — dozens of candidates — and pass them to a reranker, often a cross-encoder that reads the query and each passage together and scores relevance more accurately. Only the top few go to the model.

5. Generation with citations

The chosen chunks go into the prompt, clearly delimited and labelled with IDs, with instructions such as: answer using only these sources, cite the IDs you used, and say so if the answer is not present.

<source id="3" title="Refund policy, section 2">
Refunds are available within 30 days of purchase...
</source>

Keyword, vector, or both

BM25, formalised in the probabilistic relevance framework described by Robertson and Zaragoza, scores documents by how often query terms appear (with diminishing returns), how rare those terms are across the collection, and document length. It is fast, needs no training, and is excellent at exact matches: names, error codes, SKUs, legal clause numbers.

Vector search matches meaning. “How do I end my plan?” finds a passage about cancelling subscriptions despite sharing no keywords. But embeddings can blur exact identifiers, struggle with rare domain terms, and treat “supports X” and “does not support X” as near neighbours.

Hybrid search runs both and merges the ranked lists — reciprocal rank fusion is a simple, popular method. Because the two approaches fail in different ways, hybrid retrieval is a strong default. Metadata filters (date, product, permissions) are often just as important as the ranking method.

How RAG fails

Most bad RAG answers are retrieval failures wearing a generation costume.

Bad chunks. Tables split from their headers, answers split across two chunks, boilerplate navigation text indexed as content. Inspect your chunks by eye; it is humbling and fast.

Retrieval misses. The relevant passage exists but is not in the top results — vocabulary mismatch, an identifier embeddings cannot handle, or a question that needs information from several documents. Measure this separately: for a set of questions with known relevant chunks, how often do they appear in the top k?

Retrieving the wrong thing confidently. Semantically similar but incorrect passages — the old version of a policy, the neighbouring product — look relevant to both the retriever and the model.

The model ignoring context. It answers from training-time memory, contradicting the sources in front of it.

The model over-trusting context. The opposite: it repeats whatever was retrieved, including outdated or wrong text, or text that was merely nearby. Grounding moves the accuracy problem to your corpus quality.

Unsupported citations. A model may cite a source that does not actually support its claim. Citations enable checking; they are not a check.

Prompt injection. Retrieved documents are untrusted input. A web page or uploaded file can contain text that looks like instructions. Treat retrieved content as data, and be especially careful when the model can take actions.

Building it well

• Start simple: good chunking, hybrid retrieval, a reranker if needed.
• Build a small evaluation set of real questions with known answers and sources.
• Evaluate retrieval (did the right chunks come back?) and generation (is the answer correct and supported?) separately, so you know which half to fix.
• Log queries, retrieved chunks and answers in production, and read them.

Larger context windows have not made RAG obsolete. Stuffing everything into the prompt costs more, adds latency, and models do not use all positions in a long context equally well. Selecting the right few passages remains valuable — and it remains the part worth most of your engineering attention.