How to measure a RAG system
Most retrieval systems ship with no number attached to them. Then quality gets debated in meetings, changes get argued from intuition, and nobody can say whether last month was better than this one.
Key takeaways
A RAG system has two failure modes and two families of metrics. Retrieval is measured with recall@k, MRR and NDCG. The answer is measured with groundedness and answer relevance.
recall@k is the first number to look at. If the passage that mattered is not among the k retrieved, nothing downstream can recover, and no model swap will help.
The evaluation set does not need to be large. Thirty to fifty real questions with their correct answers, written by someone who knows the domain, already separates a working system from a broken one.
The metric that decides operational cost is cost per query. It is the one that turns a working pilot into an unaffordable production system.
What each number catches
| Metric | What it measures | What a bad score means |
|---|---|---|
| recall@k | How often the relevant passage is among the k retrieved | Search is failing. Look at chunking and hybrid search. |
| MRR | How high up the first relevant passage appears | It finds it but buries it. You need reranking. |
| NDCG | Ranking quality when several passages are relevant | The order is wrong and the model reads the weaker evidence first. |
| Groundedness | How much of the answer is supported by what was retrieved | The model is inventing beyond the evidence. Prompt and refusal policy. |
| Answer relevance | Whether the answer addresses the question asked | It answers something adjacent. Usually query rewriting. |
| Refusal rate | How often it says it does not know | Zero is a warning sign, not an achievement. |
| Cost per query | What one answer costs at the model provider | It works and you cannot afford it at real volume. |
How to build one in a day
- Take real questionsFrom support tickets, chat logs or the sales inbox. Invented questions test the system against your imagination, which is not the distribution it will face.
- Write the correct answerBy someone who knows the domain, not by the model. This is the part that cannot be automated and the reason most teams skip it.
- Mark the source passageWhich document and which section contains the answer. This is what makes recall@k computable rather than a matter of opinion.
- Include the questions with no answerTen to fifteen percent of the set should be questions the documents do not cover. Without them you cannot measure whether the system knows how to refuse.
What numbers are reasonable
There is no universal target, because it depends on how the answer gets used. A system that suggests an answer to a human agent can live with recall@5 around 0.8: the agent catches the misses. A system that answers a customer directly and unsupervised needs to be higher, and needs a refusal policy that actually fires.
What matters more than the absolute number is the direction. The value of a baseline is that the next change can be compared against it. A team that knows recall@5 went from 0.62 to 0.81 after rechunking has an argument. A team that says the system "feels better" has a meeting.
Groundedness is the number that protects you legally and commercially. When it is low, the system is producing statements the documents do not support, and someone will eventually act on one of them.
Five things that mean nobody is measuring
- The refusal rate is zero. A system that always finds something to say is inventing in some share of cases.
- Quality is reported as a percentage with no definition of what counted as correct.
- Nobody can say what recall@k was three months ago, so degradation is invisible.
- The evaluation questions were written by the vendor, using the documents the vendor indexed.
- Cost per query has never been calculated, so nobody knows what ten times the volume costs.
Retrieval degrades on its own. New documents land, the distribution of what is indexed shifts, and recall drops with nobody watching. A quarterly re-run of the evaluation set turns that into a number that arrives before the complaint does.
In a sweep of more than eighty providers in Colombia in August 2026, none sold on these metrics. Which means you have to ask for them explicitly, because they will not appear in the proposal.
Frequently asked questions
What is a good recall@k for a RAG system?
It depends on supervision. A system that suggests answers to a human agent works at recall@5 around 0.8 because the agent catches misses. A system answering customers unsupervised needs higher, plus a refusal policy that fires when confidence is low. The more useful measure is the trend against your own baseline, not a number borrowed from a benchmark.
How many questions does an evaluation set need?
Thirty to fifty real questions already separates a working system from a broken one, which is the decision most teams need. Going to two hundred gives tighter numbers per document category and is worth it once the system is in production and changing regularly. What matters more than size is that the questions are real and that ten to fifteen percent of them have no answer in the documents.
What is groundedness?
It measures how much of the generated answer is supported by the passages that were retrieved. Low groundedness means the model is adding statements the evidence does not carry, which is what people call hallucination. It is measured separately from retrieval, because a system can retrieve perfectly and still answer beyond what it read.
Can these metrics be automated?
Retrieval metrics yes, entirely: once the evaluation set marks the source passage, recall@k, MRR and NDCG compute themselves on every change. Answer metrics can be automated with a model as judge, which is useful for tracking trends but should be calibrated against human review at least once. The part that cannot be automated is writing the correct answers in the first place.
How often should a RAG system be re-evaluated?
Monthly if the document base changes often, quarterly if it is stable, and always after any change to chunking, embeddings, the retrieval strategy or the model. The purpose is not the number itself: it is catching degradation before a customer does.