Skip to content

LLM-as-judge: when to trust it and when not to

Marcus Hale ·

Using a model to grade another model's output feels circular, and people either dismiss it entirely or adopt it without checking. Both are wrong.

Judges are fine for relative comparison, risky for absolute scores

An LLM judge asked "is this answer good, 1 to 10?" produces a number that means very little on its own. The same judge asked "which of these two answers is more faithful to the source document?" is substantially more reliable.

This matters because most of what you actually need is relative. You are not asking whether the feature is good in some absolute sense. You are asking whether revision 14 is better than revision 13. Structure your evals as comparisons and you sidestep the calibration problem for free.

Measure agreement, always

Here is the minimum bar before a judge gates anything:

  1. Take 25 examples.
  2. Have a human label them.
  3. Run the judge.
  4. Compute agreement.

Below 80%, the judge is noise. Between 80% and 90%, it is useful for flagging things a human should look at. Above 90%, you can reasonably gate on it.

The failure mode we see constantly is a team writing a judge prompt, eyeballing three outputs, deciding it looks right, and wiring it into CI. Six weeks later the judge has been quietly rejecting good changes and nobody noticed.

Judges inherit their model's biases

Some documented behaviors worth knowing about:

  • Position bias. In pairwise comparison, judges favor whichever answer came first. Randomize the order and run both directions.
  • Length bias. Longer answers score higher, independent of quality. If your rubric does not mention length, the judge will silently reward verbosity.
  • Self-preference. A judge tends to rate outputs from its own model family more highly. Use a different model as the judge than the one you are grading where you can.

None of these are fatal. All of them are embarrassing when they show up in a postmortem.

When to use a code assertion instead

A lot of what people write judges for should just be a function:

  • Is it valid JSON? Parse it.
  • Does it contain a required field? Check the field.
  • Did it leak a phone number? Run a regex.
  • Is it under the length budget? Count.

Deterministic checks are faster, free, and never drift. Use judges for the genuinely subjective part and code for everything else. Teams that do this end up with eval suites that run in seconds instead of minutes.