Draw the full path from raw item to training set, add escalation for the labels that need a second look, and put real cost numbers on 10,000 items
Day 39 of 66
You have the dials (Day 37) and a queue builder (Day 38). A field day is where you assemble the pieces into one coherent diagram, find where it breaks, and put money on it. By the end you should be able to draw the full pipeline from memory and answer, for any stage, "what happens here and what could go wrong?"
On one page, draw the flow left to right. Don't make it pretty — make it complete. Every box is a stage you can now defend.
Most labels are resolved by consensus and move straight to the training set. A small fraction shouldn't — and a pipeline without an escalation path silently ships those errors. Review tiers define when a label gets escalated and to whom.
The skill is calibrating the trigger. Escalate too little and errors leak into training; escalate too much and your senior reviewers become the bottleneck and your throughput collapses. A healthy pipeline sends only a few percent of items to review — the disputed, the high-stakes, the flagged — and resolves the rest by consensus. Review tiers are where the rare expensive errors get caught; they are not a second pass on everything.
A lead earns trust by naming where their own design breaks before someone else finds out. Walk the diagram and mark each failure mode:
| Stage | How it fails | What it costs you |
|---|---|---|
| Sampling | A category is starved or a weak spot is undersampled | The model never learns where it's worst |
| Gold | Gold is detectable, or the rate is too low to measure | Quality signal is biased or too noisy to trust |
| Consensus | N = 1 on items that needed agreement | Individual error and bias enter training undetected |
| Review | No escalation path, or everything escalates | Errors ship, or reviewers become the bottleneck |
| Training set | Eval data leaks in (contamination, Week 1) | You can no longer honestly measure if the model improved |
Estimating cost and throughput is how a design becomes a plan a director can fund. Work a realistic example for 10,000 items, assuming a steady labeling rate of about 30 items per annotator-hour.
The flat alternative — 10,000 items all at N = 3 — would be 30,000 annotations and 1,000 hours, double the cost, for quality gains on the 8,000 easy items that mostly didn't need it. Your mixed design is the same quality where it counts at half the spend. When you can say that in a planning meeting, you're doing the lead's job: trading throughput against quality with numbers, not vibes. (Keep estimates policy-safe — these are operations figures, not anything sensitive.)
An evaluator trusts the pipeline they're handed. A lead names where their own design breaks before anyone else finds out — and puts a cost number on it, so a mixed-consensus plan for 10,000 items becomes something a director can fund, not a vibe.
Say this in an interview: "I can quantify a labeling plan: 10k items at a mixed N=1/N=3 split is about 500 annotator-hours, versus 1,000 for flat 3-way — same quality where it matters at half the spend. I walk in with the failure modes already marked, too."