The distinction that decides whether a model ships — and why your eval set must never see a single gradient
Day 26 of 66
You've spent five weeks producing data that trains models — demonstrations, preference pairs, rubric scores. This week the verb changes. Evaluation doesn't train anything. It measures. And the measurement it produces is the single number a launch decision hangs on: is the new model better than the one we have now?
From where you sit, annotating and evaluating can feel like the same motion — you look at an output and you judge it. But they answer opposite questions and obey opposite rules. Confuse them and you don't just waste effort; you ship a worse model while believing you shipped a better one.
Annotation produces training data — its whole purpose is to be learned from. Evaluation measures the model to decide ship / no-ship — its whole purpose is to stay un-learned-from. The instant an eval example is trained on, it stops measuring anything honest.
A tutor rating one image is doing annotation: producing a label that feeds the system. A lead asking "did version 2 actually beat version 1, or did we just convince ourselves?" is doing evaluation. The second question is harder, and it's the one leadership pays a Human Data Lead to answer with a straight face.
Every model release runs through an eval. If the eval says "better," the model ships; if it says "worse" or "no difference," it doesn't. That makes the eval the most consequential artifact your team owns — more than any single label. A wrong rubric costs you some noisy training data. A wrong eval costs you a launch decision for an entire model.
Because the eval is a measuring instrument, it has to be held out — kept entirely separate from anything the model trained on. If even a handful of eval prompts leaked into the SFT or preference data, the model has effectively seen the answer key. It will score high on those items not because it's good, but because it memorized them. Your measurement is now a lie, and you can't tell by looking.
That leak has a name: contamination. It's the costliest mistake a data org can make precisely because it's silent — the model looks great, the eval looks passed, and the regression only surfaces in production where users feel it. A lead's job is to design the controls that make contamination structurally impossible, not to catch it after the fact.
You cannot detect a contaminated eval by reading the scores — high scores are exactly what contamination produces. The only defense is process: provenance tracking on every eval item, and a hard rule that the eval set is frozen and walled off from all training pipelines. Trust the process, never the number alone.
Once you've decided to evaluate, you choose how raters judge. The two dominant formats trade off in a way you should be able to recite cold.
| Format | The rater does… | Strength | Weakness |
|---|---|---|---|
| Pairwise comparison | "A is better than B" | More reliable — relative judgments are consistent | Gives ranking, not an absolute level |
| Likert / absolute | "This is a 4 out of 5" | Gives an absolute quality level | Drifts between raters and over time |
The research and practice both lean the same way: pairwise is more reliable. Humans are good at saying which of two things is better and bad at agreeing on what "a 4" means — one rater's 4 is another's 3, and the same rater drifts across a long session. Absolute ratings buy you a number you can track over time; pairwise buys you trustworthy relative signal. Most ship decisions are inherently relative ("is the new one better?"), which is why pairwise dominates model-vs-model evals.
Beyond format, evals differ by source. Benchmarks are fixed datasets with known answers — cheap, repeatable, but stale and easy to contaminate. Human eval is fresh judgment on new outputs — expensive, slow, but the gold standard for taste and nuance. LLM-as-judge is a model standing in for the human rater — fast and scalable, but biased in ways you'll spend tomorrow learning to control.
An annotator judges an output and moves on. A lead decides whether a given judgment is training data or measurement — and walls the eval set off so it can never become the answer key. The IC produces labels; the lead owns the one number a launch decision hangs on, and the controls that keep it honest.
Say this in an interview: "Annotation and evaluation feel like the same motion from the chair, but they obey opposite rules. I keep my eval set frozen and walled off from every training pipeline, because the instant it's trained on it stops measuring anything."