A vs B for images and clips — building comparisons with a stated basis, documenting why each winner won, and writing the instructions a rater could follow
Day 24 of 66
Field day: take this onto policy-safe work you actually touch. In Week 4 you built text preference pairs — chosen vs rejected, the fuel for a reward model. Today you build the multimodal version: two generated images (or two clips) for the same prompt, and a decision about which is better. The preference machinery is identical (the RLHF Book logic doesn't care if the output is text or pixels). What changes is that for images and video, "which is better" is meaningless until you say better on what.
Every comparison needs a stated basis: which dimension decided it. "A wins" is noise. "A wins on prompt adherence — B rendered a green car, the prompt said red" is signal a reward model can learn from. The basis is the difference between teaching the model something and confusing it.
Both outputs answer the same request. If the prompts differ, you're not measuring model preference, you're measuring prompt difficulty. Hold the prompt fixed.
Adherence is judged against the request — so the rater needs the request visible next to the images, and where relevant a reference image (e.g. the identity to preserve, the style target). Without the reference, adherence collapses back into taste. This is the single biggest difference from text pairs.
Record which dimension decided it — adherence, aesthetics, artifacts, text, or (for clips) a temporal dimension. One pair, one primary basis. If two dimensions disagree (A is prettier, B is more accurate), your dimension priority from Day 21 breaks the tie: adherence outranks aesthetics.
When the two are genuinely close, you have a choice: mark it a tie and drop it (a tie carries no preference signal — same as Week 4), or force a decision only if a clear basis exists. Don't manufacture a winner out of a coin flip; a guessed preference is poison for a reward model.
For each pair, write one line: "Chose A over B — basis: [dimension] — because [specific observable]." This is not bureaucracy. It's what lets you (a) audit the dataset later, (b) catch a rater who's secretly scoring on aesthetics when the task is adherence, and (c) hand the set to a training team who can trust the signal. A pair without a documented basis is an unverifiable claim.
Near-ties are where datasets rot. A rater who must always pick a winner will invent reasons, and those invented preferences train the reward model on noise. Build the escape hatch into the instructions: "If you can't name the dimension that decides it in one sentence, mark it a tie." Ties dropped honestly beat preferences faked confidently.
This is the lead deliverable — the document a rater follows so 100 people produce the same comparisons you would. A strong visual-comparison instruction set has:
This pairs directly with your Week 4 JSONL format — a multimodal chosen/rejected set is the same shape, with the basis recorded alongside.
A rater picks A over B. A lead makes every comparison carry a stated basis — which dimension decided it — and writes the instruction doc that lets a hundred raters produce the same pairs, including the honest tie rule that keeps faked preferences out of the reward model.
Say this in an interview: "A multimodal preference pair without a documented basis is an unverifiable claim — 'A wins' is noise, 'A wins on adherence because B rendered a green car' is signal. I build the near-tie escape hatch into the instructions, because a guessed preference is poison for a reward model."