B2B SaaS · Support automation
Northwind ships AI changes with evidence
How Northwind cut support escalation rate 34% by gating every prompt change on a golden dataset built from real tickets.
Results
- Reduction in escalation rate
- 34%
- Golden examples in six weeks
- 180
- Down from 3 weeks to validate a prompt change
- 3 days
- Policy hallucinations since the gate went in
- 0
The challenge
Northwind's support assistant answered 40% of inbound tickets without a human. It also occasionally invented refund policies, and nobody found out until a customer complained. Prompt changes shipped on the same cadence as code, with no way to tell whether a change helped.
What they did
The team instrumented the assistant with the Cloudmind SDK in an afternoon. Every escalated ticket got promoted into a golden dataset with the answer it should have given. Within six weeks they had 180 examples, and a CI check that blocked any prompt change scoring below the current production revision.
“We used to argue about whether the new prompt was better. Now we run the eval, look at the number, and move on. That single change took a week of debate out of every release.”
Stop guessing whether your AI feature got better
Free for 50,000 traces a month. No credit card, no sales call, five minutes to your first trace.
Start free