Turn raw labels into per-annotator profiles, stitch two tables together, and learn the five statistics that decide whether your data is any good
Day 32 of 66
Yesterday you met the four moves. Today we go deep on the two that earn their keep: grouping and the statistics you compute once you've grouped. This pairing is what a lead does every single morning — collapse thousands of labels into a table small enough to read, then look at the numbers that tell you where to intervene.
The single most useful thing you can compute is a per-annotator profile: for each person, how many labels did they produce, what's their average score, and how much do they vary? Three numbers — count, mean, std — per annotator, and you can already spot the person who's rubber-stamping everything (high count, low spread) or the one drifting harsh (low mean).
import pandas as pd
df = pd.read_csv("labels.csv") # item_id, annotator, score, task
profile = (
df.groupby("annotator")["score"]
.agg(["count", "mean", "std"]) # 3 numbers per person
.round(2)
)
print(profile.sort_values("mean")) # harshest raters at the topgroupby("annotator") buckets the rows by person; ["score"] picks the column to summarize; .agg([...]) computes count, mean and std for each bucket at once. sort_values("mean") orders the result so the harshest annotators surface first. This four-line block is the per-annotator profile you'll build in tomorrow's exercise.Real data is rarely in one file. You'll typically have an items table (item_id, prompt, task type, who created it) and a labels table (item_id, annotator, score). To ask "what's the average score by prompt category?" you have to join them on the key they share — item_id.
pd.merge(items, labels, on="item_id") matches every label to its item by the shared item_id and produces one wide table with columns from both. Now each label row also knows its prompt category, so you can group by that category. A join is just "look up the matching row in the other table and glue the columns together."
The most useful breakdowns almost always need a join: score by task type, disagreement by prompt source, error rate by annotator seniority. The interesting dimension usually lives in a different table than the score.
The pattern is always the same: merged = pd.merge(items, labels, on="item_id"), then merged.groupby("task")["score"].mean(). Join to bring the dimension in, group to summarize along it.
You do not need a statistics degree. You need five ideas, used honestly. Everything a lead reports about data quality is built from these.
| Statistic | What it tells you | Lead's use |
|---|---|---|
| Mean & median | The center — but mean is pulled by outliers, median isn't | If mean ≫ median, a few extreme scores are skewing you |
| Std (spread) | How much labels vary around the center | High spread on one item = disagreement = guideline gap |
| Distribution shape | Are scores bunched, flat, or bimodal? | A bimodal item means two camps read the rubric differently |
| Sampling error / CI | How much your estimate could be off, given sample size | "92% ± 4%" — never report a rate without the band |
| Correlation ≠ causation | Two things move together — that's not proof one causes the other | Stops you blaming the wrong cause for a quality drop |
You met this in Week 3 with gold-set accuracy: you check a sample, not everything, so your estimate carries uncertainty. The same idea governs every number you compute this week. "Annotator C averages 2.8" on 12 labels is noise; on 1,200 labels it's a signal. Always ask how many rows are behind a number — that's exactly why count rides alongside mean in your profile.
Annotator D's quality dropped the same week you onboarded three new people. Did D get worse, or did the easy items get routed to the newcomers, leaving D the hard ones? The numbers correlate; the cause is unproven. Before you act on a finding, ask "what else changed at the same time?" A lead who confuses correlation for causation coaches the wrong person and fixes the wrong thing.
groupby("annotator")["score"].agg(["count","mean","std"]).mean and by std. Who's harshest? Who's most inconsistent?pd.merge it with labels on item_id and compute mean score by task type.An IC reads "annotator C averages 2.8" and goes to coach C. A lead reads the count beside the mean first, asks "what else changed that week," and refuses to act until correlation has been separated from cause — so they fix the right thing, not the visible one.
Say this in an interview: "I never report a rate without its sample size and confidence band, and before I act on a quality dip I rule out the confounder. A mean over twelve labels is noise, and the easy-items-got-rerouted explanation is usually the real one."