"Is our data good?" is not a vibe — it's two separate measurements, and confusing them is how data orgs fool themselves
Day 11 of 66
Last week you learned to write the rubric. This week you learn to prove whether it's working. The difference between a tutor and a lead often comes down to a single moment: someone asks "is this label set any good?" and the lead doesn't say "I think so" — the lead says a number, and what to do about it.
The trap is that "good" is two questions wearing one coat. Pull them apart now and the rest of the week clicks into place.
1. Do your raters agree with each other? That's consistency — measured by inter-annotator agreement (IAA). It tells you whether the task is well-specified.
2. Are your raters right? That's accuracy — measured against a gold set of known-correct answers. It tells you whether the labels match reality.
This is the single most important idea in the week, so we'll hit it twice. Imagine three raters all label the same blurry, artifact-ridden image "accept." They agree perfectly — 100% — and they are all wrong. High agreement, zero accuracy.
Now imagine two excellent raters who each catch real problems but apply a fuzzy guideline differently — one calls a borderline image "accept," the other "borderline." They disagree, yet both are defensible. Low agreement does not always mean low accuracy; it often means the guideline didn't decide the case for them.
"Agree + Wrong" is why you cannot measure quality with agreement alone. A team can drift into a shared misreading of the rubric, post a beautiful κ of 0.9, and quietly poison the model. Agreement measures consistency; only gold measures correctness. You need both instruments.
The instinct is to measure agreement as percent (observed) agreement: the fraction of items two raters labeled identically. Twenty items, identical on seventeen, that's 85%. Simple, intuitive — and quietly misleading.
Suppose a task has only two labels, accept and reject, and 90% of real items are accepts. Two raters who barely read the item and just stamp "accept" most of the time will agree the vast majority of the time — purely by luck, because the distribution is lopsided. Percent agreement rewards them for it. The number looks healthy; the raters are doing nothing.
The fix is to ask: how much did they agree beyond what chance alone would produce? That correction is what Cohen's kappa (κ) gives you. It subtracts out the agreement you'd expect from each rater's label frequencies and reports only the agreement that's "real." Tomorrow you'll compute it by hand; today, just hold the intuition: percent agreement does not account for chance; kappa does.
The clearest practical walkthrough — percent agreement, Cohen's kappa, and Fleiss' kappa with worked Python — is Louis de Bruijn's Inter-Annotator Agreement article. Read it as your main text this week. For the interpretation bands you'll lean on, the Fleiss' kappa reference keeps the slight/fair/moderate/substantial/almost-perfect table at hand.
You cannot fix "Agree + Wrong" with any agreement metric, no matter how clever. You need items where you already know the correct answer — carefully labeled by an expert (often you), with the reasoning written down. That's a gold set. Run your raters' labels against it and you get accuracy: how often they match the truth, not just each other.
IAA (agreement) answers "is this task well-specified enough that reasonable people land in the same place?" Gold sets (accuracy) answer "are the labels actually correct?" You'll build agreement code on Day 13 and gold sets on Day 14. A lead always reports both, because each catches a failure the other is blind to.
Notice how this connects to your Week 1 vocabulary: agreement and gold-set accuracy are the evaluation of your own label-production process. You're not measuring the model yet — you're measuring the humans (and the guideline) that feed it. If that input is noisy or wrong, no amount of model work downstream can recover it.
An IC answers "is the data good?" with a vibe. A lead refuses the question until it's split into two measurements — agreement and accuracy — and knows the trap that high agreement can mask a whole team being confidently wrong.
Say this in an interview: "When someone asks if our data is good, I don't answer with one number. I report inter-annotator agreement and gold-set accuracy separately, because a team can post a beautiful kappa and still be agreeing on the wrong answer."