Write the eval-set design doc a director would sign — and learn to spot the flattering eval that ships a worse model
Day 29 of 66
You can run a debiased judge now. But a judge with no eval set is a scalpel with no patient. The real deliverable — the thing leadership reads before a launch — is the eval-set design doc: the document that says what you measure, on what data, controlled how, scored which way, and what number decides the launch. Today you write one for a task you know cold.
An honest eval doc has six parts. Skip any one and the whole measurement is suspect.
State precisely what you are measuring, in one sentence a stranger could act on. Not "image quality" but "given a text prompt, does the generated image match the prompt's described subjects, scene, and style." Vagueness here makes every later number meaningless.
Your eval items must mirror the real distribution of work, not the easy slice. Use stratified sampling: deliberately cover each important category — by prompt type, difficulty, and any failure mode you care about — so a model can't pass by acing the common case while failing the rare-but-critical one. An eval that's all easy prompts measures nothing useful.
Spell out how the eval set stays held out: where the items came from, that they are frozen, and that they are walled off from every training pipeline. Track provenance on each item. This is the section that makes your number honest (Day 26), so write it like an auditor will read it — because one will.
Choose the format and write why. For a model-vs-model ship decision, pairwise is usually right: more reliable, and the question is inherently relative. Choose Likert only when you genuinely need an absolute level to track over time, and then accept the rater drift it brings. The justification is the point — a format with no stated reason is a guess.
Name the single number that decides ship / no-ship, and the threshold. "New model ships if its debiased pairwise win rate over the current model exceeds 55%." One metric, one bar, agreed before you run it — not chosen after the results are in.
Enough items that the result isn't noise. The intuition: a win rate from 20 comparisons swings wildly; from a few hundred it's stable. If two models are genuinely close, you need more samples to tell them apart with confidence. State roughly how many items you'll use and why — a number with no error bar is not a measurement.
The most dangerous eval isn't a broken one — it's a flattering one that looks fine. It passes, the model ships, and users feel the regression you didn't measure. This is Goodhart's law in its purest form.
When a metric becomes the target, it stops being a good metric. If everyone optimizes to the eval, the model learns to win the eval rather than to be good — and a flattering-but-broken eval will cheerfully ship a worse model. The metric and the goal quietly diverge, and the number keeps going up while quality goes down.
How a lead spots a flattering eval before it does damage:
Before you trust any eval, ask the adversary's question: "If I wanted to make a worse model look better on this eval, how would I do it?" Every answer is a hole to close. An eval you can't attack is one you can defend in a launch review.
This doc plus Day 28's llm_judge.py is the core of the Phase 1 capstone you assemble tomorrow.
An IC scores items against a metric someone handed them. A lead writes the metric — the eval-set design doc a director signs — and pressure-tests it by asking the adversary's question: how would I make a worse model look better on this? Every answer is a hole to close before the number gets to defend a launch.
Say this in an interview: "I write the eval-set design doc before anyone runs the eval: task, stratified sampling, contamination controls, format with a justification, the single ship metric and its threshold, and sample size. Then I try to game my own eval — if I can flatter a worse model past it, it's not done."