Fold the rubric and the pipeline into one program doc, make it internally consistent, add risk and mitigation, and pressure-test every decision
Day 57 of 66
Yesterday you scoped the program and mapped each part to an artifact. Today you fold the two load-bearing artifacts — the rubric (Week 2) and the pipeline / sampling / gold plan (Week 8) — into a single program document and make them agree with each other. A binder of artifacts that contradict themselves is worse than no binder; it tells a director you assembled parts without thinking about how they fit.
The rubric defines what good looks like; the pipeline defines how that judgment gets produced at quality and scale. Together they are the engine of the program — everything else (datasets, QA, calibration) is built on top of these two. Get the core right and consistent first.
Drop your Week 2 / Week 5 rubric into the program doc as the scoring definition, then check it against the program's goal. The rubric must weight the thing the program is trying to move.
Now drop in the Week 8 pipeline: stratified sampling, gold-injection rate, and difficulty-based consensus. Then check it against the rubric and the goal.
A consistent program is one where the outputs of each part are the valid inputs of the next. The rubric's verdicts feed the preference pairs; the pipeline's gold items are scored by the same rubric; the consensus rule matches the agreement target. When two parts disagree about a definition, you have a silent defect that will surface as noise in the data three weeks in.
One definition of "pass." The rubric, the gold answers, and the eval must all use the same threshold for what counts as adherence-pass. If the rubric says ≥3.5 and the gold set was built at ≥4.0, your accuracy numbers are lying.
Sampling targets the goal. If the program is about adherence failures, the pipeline must over-sample prompts where adherence is hard — random sampling wastes the budget on cases that already pass.
Consensus matches stakes. Safety items get 3-way consensus; routine adherence calls can be 1-way with gold monitoring. The pipeline's consensus rule should mirror the rubric's severity.
A director's first instinct is to ask "how does this go wrong?" Answer before they ask. List your three most likely failure modes and, for each, a concrete mitigation already built into the program.
| Failure mode | Why it happens | Mitigation (already in the program) |
|---|---|---|
| Annotators drift / agreement decays | Guidelines age, the task distribution shifts, raters tire | Weekly gold-set audit + drift alert (Week 9); calibration cadence (Week 10) |
| Eval gets contaminated | Eval prompts leak into the preference/SFT data | Held-out eval set, never sampled into training; contamination check before each run (Week 6) |
| Adherence rises but safety slips | Optimizing for compliance raises harmful compliance too | Safety gate as auto-fail in the rubric (Week 2) + safety triage on a sample (Week 11) |
"There's a risk of drift" is a worry. "We catch drift with a weekly gold-set audit and a >5-point alert that triggers a calibration session" is a plan. A lead is the person who turns every named risk into a named control. List the risk, the trigger, and who acts when it fires.
Before you trust your own decisions, check them against the canonical reference. Open the RLHF Book and skim the chapters that map to your program's core — preference data, reward modeling, and evaluation. You're not reading to learn; you're reading to challenge yourself.
Does my chosen/rejected format match what a trainer actually consumes? Am I handling ties and position bias the way the field recommends? Is my eval measuring the thing I claim, or something correlated with it? Have I confused preference data (trains) with eval data (measures)? If the book makes you flinch at a decision, that flinch is worth a revision before a director finds it.
An IC makes each artifact good on its own. A lead makes them agree: confirms the rubric's "pass" threshold, the gold set, and the eval all use one definition, so a director can't find a contradiction the team shipped by accident — and turns every named risk into a named control with a trigger and an owner.
Say this in an interview: "Before I present a program I run the joins — does each part's output cleanly feed the next, do the rubric and eval share one definition of pass. A single inconsistency between two artifacts becomes noise in the data three weeks later, so I'd rather catch it on the page."