Trace one preference click all the way through the reward model to a changed model — and learn the format the training team actually needs
Day 16 of 66
Every shift, you pick the better of two Grok outputs. In Week 1 we named that act — it produces preference data for the reward model. This week we open the box. Where does that click go? What does it physically change? By Friday you'll be able to follow one comparison from your screen to a different model, and you'll have built the dataset a training team consumes.
Your comparisons don't train the model directly. They train the grader — the reward model that scores every output the model will ever generate during RLHF. You are not labeling one image; you are calibrating the judge that will rate millions. That is why a single careless pair does outsized damage.
You already know the shape: comparisons → reward model → RLHF. Now hold each stage long enough to see the mechanism.
For a given prompt, the model generates two (or more) responses. You judge which is better. That judgment is stored as a chosen / rejected pair — not a number, a relation. "This one beat that one." Thousands of these accumulate.
A reward model (RM) is trained on those pairs with one objective: score the chosen response higher than the rejected one. After training, you hand it any response and it returns a scalar — its prediction of how much a human like you would prefer it. The RM has effectively compressed your taste into a function.
The policy (the model being improved) generates outputs; the RM scores them; reinforcement learning nudges the policy to produce higher-scoring outputs. A KL constraint tethers it to the original SFT model so it can't drift into gibberish that games the RM. The model that ships is the one that learned to please your grader without breaking.
The policy never sees your clicks. It only ever sees the reward model's scores. So every flaw in your comparisons gets baked into the RM, and the RM then propagates that flaw across every optimization step. Comparisons train the grader, and the grader trains everything. Quality at the comparison stage doesn't add — it multiplies.
You can collect preference two ways: ask "which is better, A or B?" (pairwise) or "rate this 1–5" (scalar). They feel interchangeable. They are not. Pairwise is the more reliable signal, and a lead must know why.
| Pairwise (A vs B) | Scalar (rate 1–5) | |
|---|---|---|
| The human judgment | Relative — "this beats that" | Absolute — "this is a 4" |
| Consistency across raters | High — easy to agree on direction | Low — your 4 is my 3 |
| Drift over time | Stable | Scales creep as a rater calibrates |
| What the RM wants | Exactly this | Must be converted to pairs anyway |
The reason is cognitive: humans are far better at relative judgments than absolute ones. Deciding "B is sharper than A" is fast and consistent; deciding "A is a 3.5 out of 5" forces you to invent a fixed scale in your head — and everyone's scale is different and slides as they go. This is why the canonical RLHF recipe collects comparisons, not ratings. Scalar ratings have their place (evaluation, triage), but for training the grader, pairs win.
Here's what your click becomes on disk. Reward-model trainers ingest JSONL — one JSON object per line — in this shape:
{"prompt": "...", "chosen": "...", "rejected": "..."}That's it. No score, no rank number — the relation is the data. A tie carries no signal (chosen and rejected would be indistinguishable), so ties are dropped, not stored. You'll build a file exactly like this on Day 18. A lead who hands the training team clean JSONL — and knows why every field is there — is the difference between a smooth ingest and a week of back-and-forth.
Re-read Illustrating RLHF in full now — you skimmed it in Week 1; today the reward-model section should click. Then reread InstructGPT §3 and watch specifically for how they collected comparisons (not ratings) and what their labeler setup looked like. Note one thing their pipeline does that yours at xAI doesn't.
A tutor makes a clean comparison. A lead knows the click trains the grader, not the model — so one careless pair doesn't add one error, it bakes a wrong rule into the reward model that mis-scores every output downstream. That asymmetry is why a lead defends the comparison stage harder than any other.
Say this in an interview: "I treat a preference pair as calibrating the judge, not labeling one example. A bad pair doesn't get diluted like a bad SFT sample — it teaches the reward model a rule it applies to millions of outputs, so quality there multiplies."