How to run an offline eval against last week’s traffic

Hand-written eval sets go stale within a month because the product changes and the questions change with it. Replaying real traffic does not have that problem.

What we changed

• Sample across the whole week, not the busiest hour.

• Strip PII on the way in, not on the way out.

• Keep the failures as a permanent dataset. They are the most valuable rows you will ever collect.

The full walkthrough is in the documentation.