Week 4 of 12 · The Craft

Build a Reward-Model Dataset

Turn your Week 2 rubric scores into clean chosen/rejected JSONL — with a tie-drop, a position-swap check, and a validation pass

Day 18 ~75 minutes Build Day

Day 18 of 66

Today you manufacture the fuel

You've traced the click (Day 16) and designed the collection (Day 17). Now you build the artifact a training team actually ingests: a file of chosen/rejected pairs in valid JSONL. The full preference_pairs.py is in the Try This box below — type it in, don't paste, so you read every line.

This is not a new dataset — it's your Week 2 rubric, paying off

The inputs to today's script are rubric-scored outputs — the exact scores your Week 2 rubric.py produces. The rubric gave each output a number; today you convert those numbers into the relations the reward model wants. Two weeks of work join into one pipeline: rubric → scores → pairs → grader.

What the build does, step by step

  1. Take rubric-scored outputs for one prompt. Several responses, each with a score from your rubric.
  2. Form every pair. For each combination of two outputs, the higher score is chosen, the lower is rejected.
  3. Drop the ties. Equal scores carry no preference signal — skip them. Don't fabricate a winner.
  4. Export clean JSONL. One {"prompt", "chosen", "rejected"} object per line. This is the file the trainer reads.
  5. Add a position-swap consistency check. Re-present a pair with the order flipped; a real preference survives the swap, a position-biased one doesn't.
  6. Validate the file. Re-read it line by line: every line is valid JSON, has all three keys, and chosen ≠ rejected.

The build

The "Try This — On Real Data" box below contains the full preference_pairs.py. Type it in and run it. Then point it at your own rubric-scored outputs — the ones from Week 2 — and generate at least ten real pairs.

As each pair prints, say the RM's objective out loud: "reward(chosen) > reward(rejected) — that's the whole signal." If you can't defend why the chosen one won, that pair is noise, and noise in a grader is worse than no data.

Why one bad pair becomes systematic bias

A single mislabeled image in an SFT set is one bad example among thousands — diluted. A single bad pair is different: it teaches the reward model a wrong rule ("this kind of output beats that kind"), and the RM then applies that rule to every output it scores during RLHF. One bad pair doesn't add one error — it bends the grader, and the bent grader mis-scores forever. This is exactly why the validation and swap-check steps aren't optional polish; they're the difference between fuel and contamination.

After it runs

Open prefs.jsonl in a text editor and read it as the training team will: is every line parseable? Any chosen == rejected slip through? Then run the swap check on a few pairs and note any that flipped — those are your position-bias casualties. Screenshot the valid output and the pair count; it's the start of this week's portfolio artifact.

The lead move

An annotator produces pairs. A lead ships a validated dataset — and owns the steps that make it trustworthy: the tie-drop, the position-swap check, and a line-by-line validation pass that proves every record is parseable JSON with chosen ≠ rejected. A lead hands the training team fuel, not a file they have to debug.

Say this in an interview: "I don't just generate chosen/rejected pairs — I run a swap-check to catch position bias and validate the JSONL before it leaves my hands. If I can't defend why the chosen response won, that pair is contamination, and I drop it rather than poison the grader."

Today's Takeaways