Evaluating a RAG system

RAG systems fail in two distinct places, and evaluating end to end tells you nothing about which one broke. This guide sets up separate evaluations for each.

1. Instrument retrieval as its own span

You cannot evaluate retrieval if you are not recording it.

const chunks = await trace.span({ name: "retrieve", kind: "retrieval" }, async (span) => {
  const docs = await store.search(query, { k: 5 });
  span.record({
    query,
    chunks: docs.map((d) => ({ id: d.id, text: d.text, score: d.score })),
  });
  return docs;
});

Recording the similarity score matters. Retrieval that returns the right chunk at rank 5 with a low score behaves very differently from the same chunk at rank 1.

2. Build a retrieval dataset

Take 30 real queries. For each, record which chunk IDs should come back. This is manual and takes about an hour.

npx cloudmind dataset create rag-retrieval --from-traces --filter 'span.kind=retrieval' --limit 30

Then annotate expected chunk IDs in the UI.

3. Add retrieval assertions

These are deterministic, so no judge is needed.

cloudmind.evals.register("recall-at-5", ({ output, expected }) => {
  const returned = new Set(output.chunks.map((c) => c.id));
  const hit = expected.chunkIds.some((id) => returned.has(id));
  return { score: hit ? 1 : 0 };
});

If recall@5 is below about 0.85, stop here. No amount of prompt work fixes missing context.

4. Add a faithfulness judge

Faithfulness is the generation metric with real teeth: is every claim in the answer supported by the retrieved chunks?

cloudmind.evals.registerJudge("faithfulness", {
  model: "gpt-4o",
  rubric: `Given CHUNKS and ANSWER, identify every factual claim in ANSWER.
Score 1 if every claim is directly supported by CHUNKS.
Score 0 if any claim is unsupported, even if it is true in general.
Statements of inability to answer are always faithful.`,
});

The last line matters. Without it, judges penalize "I do not have that information," which is exactly the behavior you want to encourage.

5. Calibrate the judge

Label 25 examples by hand, then:

npx cloudmind eval calibrate faithfulness --dataset rag-golden

This reports agreement between the judge and your labels. Below 85%, sharpen the rubric before gating anything on it.

6. Gate the deploy

- name: RAG evals
  uses: cloudmind-ai/eval-action@v1
  with:
    api-key: ${{ secrets.CLOUDMIND_API_KEY }}
    suite: rag-quality
    fail-under: 0.9

Troubleshooting

Recall is high but answers are wrong. The chunk is being retrieved but ignored. Check where it lands in the context; models attend unevenly and position 8 of 10 is a known dead zone.

Faithfulness is high but users are unhappy. You are measuring the wrong thing. Faithful answers to irrelevant chunks score well. Add an answer-relevance judge.

Scores swing between runs. Your judge temperature is above zero. Set it to 0.