Why your RAG pipeline gets worse in production

Test questions are written by people who know what the documents say. Real questions are written by people who do not, and they use different words.

What we changed

• Sample real queries and check what was retrieved, not just what was answered.

• Log the retrieved chunk IDs on every trace — without them a bad answer is unattributable.

• Measure how often the right document was retrieved but ignored. It is more often than you think.

The full walkthrough is in the documentation.