🪴 GoDeep
Search
← AI Engineering & Agents

Prompting, RAG or fine-tuning?

A decision framework for adapting an LLM: prompt first, retrieve for missing knowledge, fine-tune for behaviour at scale.

12 cards · 7 quiz questions · 8 min read

When a model doesn’t do what your feature needs, there are three broad ways to change that: change what you ask (prompting), change what the model can see (retrieval-augmented generation, or RAG), or change the model itself (fine-tuning). Teams often jump straight to the most impressive-sounding option. A better habit is to work out what kind of gap you have, then pick the cheapest tool that closes it.

Start by diagnosing the gap

Before choosing a technique, look at real failures and sort them into two buckets:

  • Knowledge gaps. The model doesn’t know something: your internal docs, this week’s prices, a customer’s order history, anything private or recent. It may also “know” something that has since changed.
  • Behaviour gaps. The model has the knowledge but does the task badly: wrong format, inconsistent tone, poor judgement on a narrow classification, or ignoring instructions some of the time.

This distinction does most of the work. Knowledge gaps point towards retrieval. Behaviour gaps point towards better prompts first, then possibly fine-tuning.

You can’t tell which bucket a failure falls into, or whether a fix worked, without an evaluation set. Build a small one before you change anything.

Prompting: always the first step

Prompting means clearer instructions, relevant context, a few worked examples, and an explicit output format. It has almost no setup cost and you can iterate in minutes.

Its limits are real but often overestimated:

  • Long prompts cost tokens and add latency on every request.
  • Instructions are followed probabilistically, so a rule that holds 95% of the time may not be good enough.
  • The context window caps how many examples and how much background you can include.

Even if you end up using RAG or fine-tuning, the prompting baseline tells you how much those investments actually buy.

RAG: when the problem is knowledge

RAG retrieves relevant text (from a search index, vector store, database or API) at query time and puts it in the prompt. The model then answers using that material.

It is the right fit when information is private, large, or changing. Updating knowledge means updating the index, not retraining a model, and you can show users which passages an answer came from.

The costs sit in the pipeline:

  1. Chunking and indexing documents sensibly.
  2. Keeping the index in sync with the source of truth, including deletions.
  3. Enforcing access control so users only retrieve what they’re allowed to see.
  4. Evaluating retrieval on its own. If the right passage never reaches the context, no model can use it.

Fine-tuning: when the problem is behaviour at scale

Fine-tuning continues training a model on your own examples so a behaviour lives in the weights rather than the prompt. It earns its cost when:

  • You need a consistent format, style or narrow skill that prompting delivers unreliably.
  • You run at high volume, and a smaller fine-tuned model with a short prompt could replace a larger model with a long few-shot prompt, cutting cost and latency.

Full fine-tuning of a large model is expensive, so most teams use parameter-efficient methods. The best known is LoRA (Hu et al., 2021): it freezes the pretrained weights and trains small low-rank matrices added to selected layers. That dramatically reduces trainable parameters and memory, and lets you keep many lightweight adapters for one base model.

The hidden costs are rarely the training run itself:

  • Curating hundreds or thousands of high-quality examples. Errors in the data become errors in the model.
  • An eval set to prove the tuned model beats the prompted baseline.
  • Re-tuning when the base model is upgraded or your requirements shift.
  • Constraints on where the tuned model can be hosted.

Why fine-tuning is a poor way to add facts

It is tempting to “teach the model our documentation” by fine-tuning on it. This usually disappoints:

  • Facts are absorbed unreliably; the model may recall them inconsistently or blend them with older knowledge.
  • There’s no provenance, so the model can’t cite where an answer came from.
  • Updating or removing a fact means another training run.
  • Training on unfamiliar facts can encourage confident answers where the model should be uncertain.

Research backs this up. Ovadia et al. (2023) compared unsupervised fine-tuning with RAG on knowledge-intensive tasks and found retrieval consistently came out ahead, for both previously seen and genuinely new knowledge. Treat that as strong evidence for a default, not a law: fine-tuning on well-constructed question-and-answer data can help in some settings, but it’s rarely the easiest route to accurate, current facts.

A practical decision path

1. Build a small eval set from real inputs.
2. Improve the prompt. Measure.
3. Failures about missing or changing information?  -> add RAG. Measure.
4. Remaining failures about format, style or skill,
   or prompts too long and costly at your volume?   -> consider fine-tuning (e.g. LoRA). Measure.
5. Re-run evals whenever the base model, data or prompt changes.

Combining them

These options aren’t exclusive. A common mature setup uses all three: a concise prompt sets the task, retrieval supplies current documents, and a fine-tuned model reliably answers only from the provided context, in a fixed cited format. Each tool handles what it’s good at: RAG for what the model knows, fine-tuning for how it behaves, and prompting to glue the request together.

The trade-off is operational complexity. Every layer is something to monitor, version and re-evaluate. Add layers only when your evals show the simpler setup has run out of road.

Sources