RAG vs fine-tuning: which one you need
The question usually arrives framed wrong, as if these were two routes to the same place. They solve different problems. One gives the model information it never had; the other teaches it a way of answering.
Key takeaways
RAG gives the model information it did not have at the moment it answers. Fine-tuning teaches behaviour: format, tone, domain vocabulary, output structure.
If the problem is that the model does not know your documents, fine-tuning will not fix it. Training on data that changes weekly means retraining weekly.
RAG can be audited: every answer can cite where it came from. A fine-tuned model cannot show its source, and in regulated sectors that usually ends the discussion.
The two combine. The order that works is RAG first, fine-tuning after, and only if a behaviour problem survives that the prompt could not fix.
What each one does
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What changes | What the model sees when it answers | How the model answers |
| When it helps | Proprietary, changing or large information | Stable format, tone or vocabulary |
| Updating | Reindex the new document, minutes | Retrain the model, hours or days |
| Setup cost | Low to medium | Medium to high, plus dataset curation |
| Cost per query | Higher: retrieved context is billed as input | Lower: the prompt gets shorter |
| Citing the source | Yes, with the exact passage | No |
| Main risk | Retrieval misses and the model answers on garbage | It learns a bias from the dataset and repeats it |
| How it is measured | recall@k, MRR, NDCG, groundedness | Against the base model on a held-out set |
Training to solve a knowledge problem
The typical case arrives like this: "the model knows nothing about our products, so we need to train it on our catalogue." It sounds reasonable and it almost never works. Fine-tuning shifts weights toward a distribution of answers; it does not store facts in a way you can look up. A model trained on the catalogue will sound like the catalogue and will invent part numbers with exactly the confidence it had before.
The proof is in the update cycle. If a product changes price on Tuesday, RAG reindexes the document and the answer changes that afternoon. With a fine-tuned model you curate the dataset again, retrain, evaluate and deploy. Nobody sustains that week after week.
Where fine-tuning does win is behaviour. A model that must always return the same structure, use a specific industry vocabulary or hold an unusual tone learns that far better from examples than from a two-thousand-word prompt. And because the prompt shrinks, every query gets cheaper.
How to decide without overspending
- Start with the promptA well-written system prompt with three examples solves more formatting problems than people expect, and it costs an afternoon.
- If information is missing, use RAGYour documents, policies, catalogue, history. With hybrid search and reranking before you consider anything else.
- Measure before you go furtherWithout an evaluation set you cannot tell whether the next step improved anything. recall@k for retrieval, groundedness for answer faithfulness.
- Only then, fine-tuningAnd only if what remains broken is behaviour, not knowledge. With the evaluation set already in place to compare against the base model.
Three situations and what fits
Insurer with policies that keep changing
Hundreds of documents updated by regulation, and answers that must cite the exact clause. Fine-tuning is unworkable here: it cannot show a source and it would be stale after every update.
Classifying messages into a proprietary taxonomy
Thousands of labelled examples, stable categories and a one-word output. A small tuned model is cheaper per query and faster than a large one carrying a long prompt.
Technical support with a house voice
RAG for product documentation, which changes with every release, and fine-tuning for the answer style and the fixed structure the quality team requires.
Either route without measurement is a bet. Before choosing you need a set of real questions with their correct answers and a baseline: how often the system retrieves the passage that mattered, and how often the answer stays inside what was retrieved.
It is the part no vendor in the Colombian sweep sells, verified across more than eighty companies in August 2026. It is also the only way to know whether a change improved anything.
Frequently asked questions
What is RAG, briefly?
Retrieval-augmented generation. Before answering, the system searches your documents for passages relevant to the question and hands them to the model inside the prompt. The model answers on that, and can cite where each statement came from. It does not change the model: it changes what the model sees.
When is fine-tuning the better choice?
When the problem is behaviour rather than knowledge: a fixed output structure, a very specific domain vocabulary, a tone the prompt cannot hold, or a repetitive classification task where cost per query matters. If the missing information changes over time, fine-tuning is the wrong tool.
Can you use both together?
Yes, and mature systems usually do. Order matters: RAG first, measurement second, fine-tuning only if a behaviour problem survives that the prompt could not fix. Reversed, you pay for training only to discover the problem was retrieval.
Which is cheaper?
RAG costs less to start and more per query, because retrieved context is billed as input tokens. Fine-tuning costs more to start (dataset curation, training, evaluation) and less per query, because the prompt stays short. At low volume RAG wins; at very high volume on a stable task, tuning amortises.
How do I know whether my RAG is working?
Two numbers. recall@k measures how often the passage that mattered shows up among the k retrieved: if it is low, the problem is search and no model swap will fix it. Groundedness measures how often the answer stays inside what was retrieved: if it is low, the model is inventing beyond the evidence. Without those two numbers, every quality discussion is opinion.