How to build your first LLM golden dataset

Most teams put off building an eval dataset because they imagine it means labeling a thousand examples. It does not. Fifty well-chosen examples will catch the majority of regressions you actually ship, and you can assemble them in an afternoon from data you already have.

Start with your failures, not a sample

The instinct is to take a random sample of production traffic. Resist it. A random sample is mostly easy cases, and easy cases do not regress.

Instead, pull from these three buckets:

1. Every bug report you have. Someone already did the hard work of noticing a failure. Each of those is a test case.
2. Low-confidence outputs. If you capture user feedback, the thumbs-down traces are pre-labeled failures.
3. Structurally weird inputs. Empty fields, very long inputs, non-English text, prompt injection attempts. These break things and nobody thinks to test them.

Aim for roughly 60% failure cases and 40% known-good cases. You need the good ones so you can detect when a fix breaks something that used to work.

Write the expected output, not just a label

"This answer was bad" is not a test case. What would a good answer have been?

This is the part people skip and it is the part that matters. For each example, write down the output you would have wanted. It does not need to be word-perfect, because you are going to grade with a judge rather than string equality, but the judge needs something to compare against.

{
  "input": { "ticket": "I was charged twice for my October invoice" },
  "expected": "Acknowledge the duplicate charge, confirm a refund will be issued within 5-7 business days, and do not promise a specific amount without checking the account.",
  "tags": ["billing", "refund-policy"]
}

Note that the expected output encodes a policy constraint: do not promise an amount. That is the kind of thing that regresses silently when someone rewrites a system prompt.

Calibrate your judge before you trust it

Once you have examples, you need a grader. LLM-as-judge is the usual answer, and it works, but only if you check it.

Label 20 examples by hand as pass or fail. Run the judge on the same 20. Measure agreement. If the judge agrees with you less than about 85% of the time, your rubric is too vague, not your model. Sharpen the rubric and try again.

We have watched teams gate deploys on judges with 60% agreement, which is barely better than a coin flip and actively harmful because it manufactures false confidence.

Grow it from incidents

The dataset is not a one-time project. Every time something goes wrong in production, the fix has two parts: change the prompt, and add the failing case to the dataset. Do that consistently for six months and you end up with a suite that encodes everything the feature has ever gotten wrong.

That is the whole trick. It is not sophisticated, it just requires doing it every time.