Skip to content

What a good eval suite looks like for a support chatbot

Priya Raman ·

The temptation is to grade helpfulness with another model and call it done. The graders that actually catch regressions are duller than that, and cheaper.

What we changed

  • Did it cite a real article ID? A string check, and it catches most hallucinations.

  • Did it refuse to give a refund it has no authority to give? A rule, not a rubric.

  • Is it under 120 words? Long answers are how quality problems announce themselves.

The full walkthrough is in the documentation.

What a good eval suite looks like for a support chatbot | Cloudmind