Evaluations
Evals that run on every deploy
Most teams evaluate once, before launch, and never again. Cloudmind runs your suite against real traffic on every change, so a regression is caught by a build rather than by a customer.
How teams use it
Graders you write
A grader is a function. Write it in Python or TypeScript, or use a model-graded rubric where the answer is genuinely subjective.
Run against production traffic
Replay last week’s real requests against a candidate change, so the eval set matches what users actually ask.
Per-change scores
Every run is tied to a prompt version and a model, so "which change moved this?" has one answer.
Fails the build
Wire a threshold into CI and a regression stops the deploy, the same way a failing test does.
“We used to find out a prompt change was bad from a support ticket three days later. Now it fails in CI before it ships, and the eval names the twelve requests it broke.”
Find the regression before your users do
Free for 50,000 traces a month. Bring your own graders.
Start free