The decisions a lead makes before a single comparison is collected — and why DPO raises the stakes on every one of them
Day 17 of 66
A tutor receives a comparison task and judges it. A lead designs the task — and a dozen decisions are baked in before any tutor sees it. Get them right and the dataset is clean signal; get them wrong and you've quietly poisoned the grader. These are the levers you'll own.
Two responses → one pairwise judgment. Four responses → you can collect a ranking and expand it into six pairs (every combination). More responses per prompt squeezes more comparisons out of each one — efficient — but ranking four images is cognitively heavier than picking between two, and fatigue erodes quality. The common sweet spot is a small ranking (4–7) that a rater can hold in their head at once.
A full ranking is data-dense but slower and noisier at the edges (the middle of a ranking is where raters waver). A single A-vs-B is fast, clean, and the most reliable unit of preference. Lead's call: rank when responses are cheap and you need volume; pairwise when the judgment is hard and you need signal you can trust.
A reward model needs thousands of pairs to learn a stable preference, not dozens. There's no magic number — it scales with task difficulty and how subtle the preference is — but "we collected 40 comparisons" is not a dataset, it's a pilot. You'll write a defensible target on Day 19.
When two responses are genuinely equal, a forced pick is a coin flip — pure noise injected into your grader. The right move is to let raters mark a tie and then drop ties from the training set. But you must collect the tie rather than forcing a fake winner: a high tie rate is itself a signal that your prompts aren't discriminating, or your model's two outputs are too similar to be worth comparing.
Raters (and LLM judges) drift toward whichever response is shown first — or sometimes second. If "A" wins more often simply because it's on the left, your RM learns "left is good," which is worthless. Control it by randomizing or swapping the order and, where you can, checking that a rater's verdict survives a swap. You'll build exactly this position-swap check on Day 18. We'll meet position bias again in Week 6 with LLM judges — it's the same enemy.
This is where Week 2 returns. If you ask ten raters "which is better?" with no definition, you get ten private notions of "better" averaged into mush. The fix is the same artifact you built in Week 2: a rubric, written into the labeler instructions, so "better" means the same dimensions, in the same priority order, to everyone.
Your Week 2 rubric isn't a separate artifact from this week's pairs — it's the spec that makes the pairs consistent. That's the through-line you'll formalize on Day 20.
Until now, your pairs trained a reward model, and the reward model trained the policy — two steps of indirection. Direct Preference Optimization (Rafailov et al., 2023) collapses that.
With a reward model in between, a noisy pair gets partly averaged out across thousands of others before it touches the policy. With DPO, the pairs are much closer to the final model — there's no RM smoothing things over. Pair quality becomes even more central. The cleaner your chosen/rejected data, the more DPO rewards you; the noisier it is, the more directly that noise lands in the shipped model. You don't implement DPO — but you must know that the modern recipe makes your design decisions matter more, not less.
A tutor judges the comparison task they're handed. A lead designs it — responses per prompt, ranking vs pairwise, the tie rule, the position-swap control, the rubric written into the instructions — a dozen decisions baked in before any rater clicks. And a lead knows DPO raises the stakes on every one, because with no reward model to average out noise, design flaws land closer to the shipped model.
Say this in an interview: "I make the collection-design calls before a single comparison is gathered — order randomization, tie handling, how the rubric defines 'better.' Under DPO there's no reward model smoothing things over, so those design decisions matter more, not less."