The three dials that set the quality–throughput trade of an entire annotation operation
Day 37 of 66
Yesterday you saw the whole flow. Today we go deep on the three stages where a lead spends real money and buys real quality: sampling (which items), consensus (how many annotators per item), and gold (how you measure that it's working). Every one of these is a trade-off, and naming the trade-off out loud is what separates a designed pipeline from a guessed one.
You almost never label everything — you can't afford to. So sampling is a decision about what the model learns about. Pure random sampling sounds fair, but it lets your largest category swamp the queue and starves the rare ones.
Stratified sampling splits the pool into groups (strata) by some property — task type, prompt category, difficulty band — and samples a deliberate amount from each. This guarantees coverage: every category the model needs to learn shows up, even the rare ones.
Example: your pool is 80% "ordinary quality" items and 20% "safety-adjacent." A random 200-item sample gives you ~40 safety items by luck. Stratifying lets you say "100 quality, 100 safety" on purpose — because safety errors are far more expensive than their natural frequency suggests.
What you sample is what you teach. Oversample a known weak spot (say, text rendering inside generated images) and the model gets more signal exactly where it's failing. Sampling is one of the highest-leverage, least-visible decisions you make — and it's invisible to the annotators, who only see the queue you built.
Consensus is the number of independent annotators who label a single item. With N = 1, one person's judgment is the label — fast and cheap, but a single mistake or bias goes straight into the training set undetected. With N = 3 and a majority vote, two of three must agree, which catches individual error and surfaces genuine ambiguity (a 1-1-1 split is a flag that your guideline is unclear).
More consensus = higher quality but lower throughput. So you don't apply it flatly — you spend it where it pays. Easy, low-stakes items get N = 1. Hard or safety-critical items get N = 3 (majority). The principle: an error's cost, not its frequency, decides how many eyes it gets.
This isn't a free choice — it's a budget allocation. Watch what 3-way consensus does to throughput on a fixed number of annotator-hours.
| Design | Annotations needed for 1,000 items | Relative cost | Quality |
|---|---|---|---|
| All N = 1 | 1,000 | 1.0× | Baseline — single point of failure per item |
| All N = 3 | 3,000 | 3.0× | High — but you label 3× fewer items per hour |
| Mixed: 800 easy @ N=1, 200 hard @ N=3 | 800 + 600 = 1,400 | 1.4× | High where it matters, cheap where it doesn't |
The mixed row is the lead's answer. The same annotator budget that would give you only ~470 fully-triple-reviewed items instead covers all 1,000 — with three eyes on exactly the 200 items where a mistake is expensive. That reallocation is the design.
Gold items are items whose correct answer you already know, mixed secretly into the live queue. An annotator labels them like any other item; you compare their answer to the known truth. The running accuracy on gold is your real-time read on quality — per annotator, per task, per week.
You inject gold at a target rate — commonly around 10%. Too little (1%) and you don't get enough gold hits to estimate accuracy with any confidence; too much (30%) and you're paying to re-label things you already know instead of getting new labels. Ten percent is a typical balance: enough signal to catch a drifting annotator quickly, cheap enough not to dominate the budget.
If annotators can spot the gold items, they'll be extra careful on those and lazy on the rest — and your measurement becomes a flattering lie. So gold must be indistinguishable from real work and shuffled in randomly, never clustered at the start or visually marked. The moment gold is detectable, it measures "how well do they do when watched," not "how good is the data." You'll implement the shuffle that hides it on the build day below.
For one real task you know, write the three settings as a single design statement — this is exactly what you'll encode in code tomorrow:
Three sentences. But each one is a defensible decision with a stated cost — and being able to say the cost out loud ("3-way consensus triples the per-item price, so I only spend it on safety") is the difference between a lead and someone repeating a default.
An annotator labels whatever the queue serves. A lead sets the three dials — sampling, consensus, gold — and can name the trade-off behind each: triple consensus triples the per-item price, so it's spent only where an error is expensive, not where it's merely frequent.
Say this in an interview: "I allocate consensus by cost-of-error, not by frequency — N=1 on cheap items, N=3 only on the safety-adjacent ones. That mixed design buys flat-3-way quality where it counts at a fraction of the budget."