RAG evaluation: what actually matters

When a RAG system gives a bad answer, the instinct is to blame the model. In our experience it is retrieval about three-quarters of the time, and you cannot tell which without measuring them separately.

Split the evaluation in two

A RAG pipeline has two failure surfaces:

Retrieval: did the right chunks come back? This is measurable with classical IR metrics and does not need a model to evaluate.

Generation: given those chunks, was the answer faithful to them? This needs a judge, but it is a much narrower question than "was the answer good."

Evaluating end to end conflates the two. A faithful answer to the wrong chunks and an unfaithful answer to the right chunks both score badly, and you learn nothing about which one you have.

Retrieval metrics worth tracking

• Recall@k: is the chunk containing the answer in the top k? This is the single most useful number, and if it is low nothing downstream can save you.
• MRR: how far down was it? Position matters because models attend unevenly across a long context.
• Chunk precision: what fraction of retrieved chunks were relevant? Low precision means you are paying for tokens that add noise.

You need a small labeled set for this: query, plus which chunks should have come back. Thirty queries is enough to see obvious problems.

Generation metrics

Once retrieval is decent, grade generation on:

• Faithfulness: is every claim in the answer supported by the retrieved chunks? This catches hallucination directly and is the one to gate on.
• Answer relevance: does it address the question that was asked, rather than something adjacent?
• Completeness: did it use the relevant chunks it was given, or ignore half of them?

Faithfulness is the metric with real teeth. An unfaithful RAG answer is worse than no answer because it comes with the implicit authority of being sourced.

The diagnostic that saves the most time

When an answer is wrong, look at the retrieved chunks first.

If the answer was not in them, it is a retrieval problem: check chunking strategy, embedding model, query rewriting, or whether the document is even in the index.

If the answer was in them and the model still got it wrong, it is a generation problem: check the prompt, whether the relevant chunk was buried at position eight, or whether the context is simply too long.

Doing this consistently is worth more than any single metric, because it stops you from tuning the prompt for two days to fix an indexing bug.