Week 4 of 12 · The Craft

The Whole Loop in Your Hands

Synthesize Week 4, prove it with a self-quiz, and see your Week 2 rubric and Week 4 pairs as a single pipeline

Day 20 ~60 minutes Review

Day 20 of 66

What you now know

This week you stopped producing preference data and started designing it. Say the spine back without looking:

The Week 4 spine

The recipe in depth: comparisons → reward model predicts human preference → RLHF optimizes the policy against it, with a KL constraint to the SFT model.
Comparisons train the grader, so quality compounds — one bad pair becomes a wrong rule applied to everything.
Pairwise beats scalar: relative judgments are easier and more consistent than absolute scores.
The trainer's format is chosen/rejected JSONL; ties carry no signal and are dropped; position bias is controlled by swapping order.
DPO trains the policy directly on pairs — no separate RM — making pair quality even more central.

Self-quiz (no notes)

Answer out loud or in writing. If any answer is shaky, reread that day before Week 5.

Your preference click doesn't touch the policy directly. What does it train, and what does that then train?
Why is "pick the better of two" more reliable data than "rate each 1–5"?
Two responses are genuinely equal in quality. What do you do with that comparison, and why?
What is position bias, and what's the one control that catches it?
DPO removes the reward model. Why does that make your pair quality matter more, not less?

Answer key (peek after you try)

1) It trains the reward model (the grader); the RM then trains the policy via RLHF — two steps removed, which is why pair quality compounds. 2) Humans make relative judgments consistently but absolute ones drift — your 4 is my 3, and scales creep over time. 3) Mark it a tie and drop it from training — a forced pick is noise; never fabricate a winner. 4) Order swaying the verdict; control by swapping/randomizing order and keeping only verdicts that survive the swap. 5) With no RM to average out noise, the pairs sit closer to the final model, so noise lands more directly in what ships.

Week 2 rubric → Week 4 pairs: one pipeline

The most important thing to leave this week with is that these are not separate skills. They're one assembly line, and you now own two stations on it.

The line, end to end

Rubric (Week 2) — defines what "good" means: weighted dimensions, safety gate, worked examples.
Scores — applying the rubric to outputs produces a number per response.
Pairs (Week 4) — higher score becomes chosen, lower becomes rejected; ties dropped; exported as JSONL.
Reward model — learns reward(chosen) > reward(rejected) — your rubric's judgment, now a function.
Policy (RLHF / DPO) — optimized to score high on that grader. The shipped model's taste traces straight back to your rubric.

Why this is the leadership view

A tutor sees a comparison task. A lead sees the whole line and knows that a vague rubric in step 1 becomes inconsistent pairs in step 3, a confused grader in step 4, and a worse model in step 5. The leverage is upstream. When you fix the rubric, you fix every step after it — which is exactly why Week 2 came before Week 4. That's the data-centric instinct from Week 1, now concrete.

Close the week

Wrap-up

Before Week 5

Self-quiz answers all solid (or reread the weak day)
Your prefs.jsonl validates: every line parseable, all keys present, chosen ≠ rejected
Week 4 summary written in your own words, naming the full rubric → pairs → RM → policy line
One sentence: where you'd target the next 1,000 comparisons and why

Up next — Week 5: Multimodal Data (Your Edge)

You'll extend everything from text to image and video — the home turf most data leads never had. The rubric and pairs you built become multimodal: prompt adherence, aesthetics, artifacts, text rendering, and the temporal dimensions that make video the hardest preference data there is. This is the week your background becomes a leadership advantage.

The lead move

A tutor sees a comparison task. A lead sees the whole assembly line — rubric → scores → pairs → reward model → policy — and knows the leverage is upstream: a vague rubric in step one becomes inconsistent pairs, a confused grader, and a worse shipped model. Fix the rubric and you fix every step after it, which is exactly why Week 2 came before Week 4.

Say this in an interview: "I don't treat the rubric and the preference pairs as separate jobs — they're one pipeline, and the shipped model's taste traces straight back to how I defined 'good' in the rubric. When the model is wrong, I look upstream at the data that taught it, because that's where one fix propagates to every step."

Week 4 Takeaways

You can trace a preference click through the reward model to a changed policy — and name the KL constraint.
You built a real chosen/rejected JSONL dataset with tie-dropping and a position-swap check.
You can design collection: weak-spot targeting, sampling, labeler instructions, and scaling controls.
You see rubric (W2) → pairs (W4) as one pipeline — and know the leverage is upstream.