A strong LLM can judge as well as a human — until you learn the three ways it lies, and why you measure it before you trust it
Day 27 of 66
Human eval is the gold standard, but it doesn't scale. You can't put a person in front of every output of every candidate model on every release. So the field reached for the obvious shortcut: ask a strong model to play the rater. Show it two outputs, ask which is better, and collect verdicts at a thousand times the speed of a human panel. This is LLM-as-judge, and it is now how most large evals actually run.
The natural objection — "a model can't possibly judge as well as a person" — turned out to be wrong, with a caveat that is the whole reason this lesson exists.
In Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), a strong judge — GPT-4 — reached over 80% agreement with human raters. That's the same level humans agree with each other. A good judge is, by that measure, as good as a person. But the same paper measured exactly where the judge goes wrong — and those failure modes are systematic, not random.
These aren't quirks; they're predictable distortions that show up across judge models. A lead who can't name them shouldn't be allowed to trust a judge. Memorize all three.
The judge favors whichever answer sits in a particular slot — often the first one shown — regardless of content. Swap A and B and the verdict can flip. This means a single comparison tells you as much about presentation order as about quality.
The judge favors the longer answer, mistaking length for thoroughness or quality. A padded, waffly response beats a tight, correct one. If your candidate model happens to be chattier, the judge will reward it for the wrong reason.
A judge tends to favor outputs from itself or its own model family. Let GPT-4 judge a contest that includes GPT-4's own answers and it tilts toward them. The judge is grading on a curve that flatters its relatives.
Beyond the three biases, judges are weaker on tasks needing real reasoning — math, careful multi-step logic, fact-checking. The judge can sound confident while being wrong about whether an answer is actually correct.
Every bias has a known mitigation. The discipline is applying all of them by default, not reaching for them after a result looks suspicious.
| Bias | Mitigation |
|---|---|
| Position | Swap the order and require the judge to agree with itself; discard verdicts that flip |
| Verbosity | Control for length — match or normalize answer length, or penalize unjustified padding |
| Self-enhancement | Never let a model judge its own family; use a neutral third-party judge |
| Limited reasoning | For reasoning-heavy tasks, fall back to human eval or give the judge reference answers |
Always measure judge-vs-human agreement before you trust the judge. Take a sample your team has already labeled by hand, run the judge on it, and compute how often it agrees with your humans. If agreement is high, the judge has earned the right to scale your eval. If it's low, the judge is a random-number generator with a vocabulary — no amount of speed makes that useful. This is the same logic as validating any rater against gold (Week 3), applied to a model.
An eval that runs on a biased, unvalidated judge will happily report that a worse model is better — and ship it. The lead who can say "our judge agrees with humans 84% of the time, we swap order to kill position bias, and we never let Grok judge Grok" is the one leadership can actually rely on. The biases are not trivia; they are the difference between an eval that protects launches and one that sabotages them.
An evaluator runs the judge and reports its verdicts. A lead measures the judge first — naming position, verbosity, and self-enhancement bias by name, then proving judge-vs-human agreement before letting a single verdict count. A fast judge nobody validated is a random-number generator with a vocabulary.
Say this in an interview: "I never trust an LLM judge until I've measured its agreement with our human labels — same logic as validating a rater against gold. And I design the eval to kill the known biases by default: swap order to catch position bias, control for length, and never let Grok judge Grok."