Measure correctness with known-answer items, hide them to catch drift, and estimate quality from a sample without checking everything
Day 14 of 66
Kappa told you whether your raters agree. It cannot tell you whether they're right — remember the "Agree + Wrong" quadrant. The instrument for correctness is the gold set: a small collection of items where you already know the correct label, with the reasoning written down.
A gold item is one an expert (often you) has labeled carefully and defensibly, with a one-line rationale: "this is a reject because the hands have six fingers — a technical-quality failure under rule 3." The rationale is non-negotiable. A gold answer without a reason isn't gold; it's just one more opinion. When a rater disagrees with gold, the rationale is what lets you say "the rater is wrong" versus "the gold is wrong, fix the gold."
Run any rater's labels against the gold set and you get accuracy: how often they match the known-correct answer. That's the number that catches a confident, consistent team that has all drifted into the same mistake.
Consensus = what most raters said (a majority vote). Gold = what's actually correct (an expert's documented answer). Consensus can be confidently wrong; gold is your ground truth. Use consensus to resolve everyday disagreements at scale, but use gold to check whether the consensus itself is any good.
A gold set you only run as a one-time test goes stale the moment guidelines or annotators change. The operational move is to inject gold items invisibly into the normal work queue — "honeypots." Raters can't tell them apart from real work, so their gold accuracy becomes a continuous, honest measure of quality you can watch over time.
Every gold item you inject is throughput you spend on measurement instead of production. Too low and your accuracy estimate is noisy; too high and you're paying to re-label known answers. ~5–10% is the usual sweet spot — but justify your number by how fast the task drifts and how costly an error is. High-stakes safety work earns a higher rate.
You'll never hand-check 50,000 labels. You don't have to. Audit a random sample, measure its accuracy, and you have an estimate of the whole set's accuracy — with an honest margin of error attached.
A point estimate without an error bar is a guess wearing a lab coat. When you report "this label set is 88% accurate," immediately attach the sample size and the range — "n = 150, roughly ±5 points." It tells leadership how much to trust the number, and it stops anyone from over-reacting to a wobble that's inside the noise. You'll formalize the confidence-interval math in Week 9; today, internalize the intuition.
Keep all of this policy-safe: gold sets for sensitive tasks live behind the same access controls as the real data, and honeypot answers stay confidential — if raters learn the gold, the measurement is dead.
An IC labels gold items well. A lead builds the instrument: gold with written rationale, hidden as honeypots in the live queue to catch drift continuously, and a sampling plan that reports accuracy as a range with an error bar — not a bare point estimate.
Say this in an interview: "I don't just check quality once — I inject gold as honeypots at a 5–10% rate so accuracy is a continuous drift alarm, and I never report a number without its sample size and error bar, because a point estimate without a range is a guess in a lab coat."