Production monitoring and alerting

Offline evals catch regressions you introduced. Monitoring catches drift you did not.

Sample production traffic for quality

Grading every production trace with a judge is expensive and unnecessary. Sample.

npx cloudmind monitors create quality-support \
  --eval faithfulness \
  --sample 0.05 \
  --filter 'metadata.feature=inbox-summary'

Five percent is usually enough to see a real shift within a few hours at moderate volume. Raise it right after a release and lower it once things look stable.

What to alert on

Eval score drop. The rolling 24-hour score falling more than 10% below the trailing 7-day baseline. This is the alert that matters most.

Cost anomaly. Token spend deviating from its trailing baseline, broken down by feature and prompt version. Catches retry loops and context bloat.

Latency drift. p95 above your target. Usually means a provider issue or a context window that has quietly grown.

Error and refusal rate. Rate limits, timeouts, content filter blocks. Refusal rate spiking often means a prompt change made the model overly cautious.

Setting thresholds that do not cry wolf

Start looser than feels right. An alert that fires twice a week gets muted within a month, and a muted alert is worse than no alert because it creates false confidence.

Use relative thresholds against a trailing baseline rather than absolute numbers. Absolute thresholds break the first time traffic patterns shift.

Routing

npx cloudmind monitors notify quality-support --slack '#ai-quality'
npx cloudmind monitors notify cost-spike --pagerduty 'ai-platform'

Quality alerts go to a channel where someone will look at them during business hours. Only page for things that need action within the hour, which in practice means cost anomalies and hard error rates, not score drift.

Responding to a quality alert

1. Open the monitor. Look at the failing examples, not the aggregate.
2. Check what changed: prompt revision, model version, index update, upstream data source.
3. If a prompt change caused it, roll back the label. That is a 60-second fix.
4. Add the failing examples to your golden dataset so the regression cannot recur silently.

Step 4 is the one people skip, and it is the one that compounds.