Aim collection at the model's weak spots, write the labeler instructions you'd ship, and hold quality while scaling to 10k pairs
Day 19 of 66
You can produce clean pairs now. A lead's job is bigger: deciding which pairs are worth collecting, writing the instructions a team will follow, and keeping quality intact as the count climbs from ten to ten thousand. Today is the operational view — the decisions you'd defend to a training team.
The instinct is to collect comparisons everywhere. That wastes the budget. If the model already nails landscape prompts, a comparison between two great landscapes teaches the grader almost nothing — both are good, the preference is a coin flip, and you've spent a rater's attention on a tie.
Signal lives where the model struggles — text rendering, hands, multi-subject scenes, fast motion in video. Target your collection there. The most useful pair isn't "great vs terrible" (obvious, low information) — it's "mediocre vs slightly-less-mediocre on a hard prompt," where a human's judgment teaches the grader a distinction it can't yet make. Spend comparisons where the model is wrong, not where it's already right.
How you choose prompts to compare on is itself a design. A few defensible moves:
This is the artifact that makes ten raters produce one dataset instead of ten. It's your Week 2 rubric, retargeted for comparison. Ship it with the task, not after.
Ten pairs you check yourself. Ten thousand pairs from a team is a different problem — quality doesn't survive scale on its own. Three controls hold the line:
| Control | What it is | Why it holds quality at scale |
|---|---|---|
| Gold pairs | Pairs with a known, expert-agreed correct winner, hidden in the stream | Catches raters (or whole batches) drifting from the standard — measurable, live |
| Agreement checks | Same pair judged by 2–3 raters; track how often they agree | Low agreement = a guideline gap, not a people problem (Week 3's lens) |
| Calibration | Periodic sessions where raters compare verdicts on shared pairs | Pulls the whole team toward one definition of "better" before drift compounds |
At 10k pairs you can't read every one — so you build the system that tells you whether the unread ones are good: a gold-pair pass-rate you watch weekly, an agreement number per task type, and a calibration cadence. You're no longer judging pairs; you're running the operation that keeps the pairs trustworthy. That's the Week 4 → Phase 2 bridge. (Weeks 8–9 turn these controls into a full pipeline and quality-ops loop.)
Everything here is generic data-collection design. Don't write down or reproduce any internal xAI prompts, model outputs, weak-spot lists, or thresholds in your portfolio — use synthetic or clearly hypothetical examples. The method is yours to keep; the proprietary data is not.
A tutor judges whichever pairs land in their queue. A lead decides which pairs are worth collecting at all — aiming the budget at the model's weak spots and close calls, not easy wins — then builds the system that vouches for 10k pairs nobody can read by hand: gold pairs, agreement checks, calibration. You stop judging pairs and start running the operation that keeps them trustworthy.
Say this in an interview: "I spend comparisons where the model is weak, not where it's already right — a close call on a hard prompt teaches the grader a distinction it can't make yet. At scale I don't read every pair; I watch a gold-pair pass-rate and per-task agreement, so the system tells me whether the data I can't read is good."