Build the analysis a lead runs every week: a per-annotator profile, the widest-disagreement items, and a one-screen quality summary
Day 33 of 66
Two days of concept; today you build. The "Try This" box below contains analyze_labels.py — type it in, don't paste, so you read every line. Then run it against a real or realistic label CSV and read what it tells you. By the end you'll have done the exact analysis a Human Data Lead opens every Monday.
If you can't pull a label file from work, hand-write a small labels.csv with columns item_id, annotator, score, task and ~30 rows across 3 annotators — deliberately make a couple of items high-disagreement. Analyzing data you constructed teaches you what the numbers should say, which is the best way to learn to read them.
A weekly label analysis isn't a data dump — it answers three questions, in order. Hold these in mind as you build; the code in the Try This box maps to them one-to-one.
groupby("annotator")["score"].agg(["count","mean","std"]). Count flags who's carrying load; mean flags who's drifting harsh or soft; std flags who's erratic. This is your people view.
Group by item_id and find the items with the highest std — where annotators disagreed most. As you learned in Week 5, disagreement is data: the widest-spread items are precise pointers to where your guideline is silent or contradictory. These are the first things you fix.
Overall count, mean, the share below the pass bar, and the top-5 disagreed items — small enough to fit on one screen and paste into a message. A lead manages from a summary, not a 10,000-row table.
This is the load-bearing insight of the whole build. The items where annotators scatter most are not "bad annotators" — they're under-specified cases. Sort by std descending, take the top 5, and you have a ranked to-do list for your next guideline revision. You're turning a spreadsheet into a roadmap.
The Try This box below contains the full analyze_labels.py. It loads the CSV, prints the per-annotator profile, then prints the most-disagreed items sorted by std. Run it; read the output as if you had to act on it tomorrow.
As you read the disagreement hot-list, write one sentence per top item: "Annotators split on item X because the guideline doesn't say ___." That sentence is the deliverable — the table is just how you found it.
Add a quick chart. A few lines of matplotlib — import matplotlib.pyplot as plt; df["score"].hist(bins=5); plt.savefig("dist.png") — gives you the score distribution. Is it bunched at 4–5 (rubber-stamping?), flat (no consensus?), or bimodal (two camps?)? The shape tells you something the mean hides. matplotlib is enough — you don't need fancier libraries.
analyze_labels.py from the Try This box and run it on your CSV.score and describe the distribution's shape in one line.An IC sees high-std items and assumes the annotators are sloppy. A lead sorts by std descending and reads the top five as a ranked guideline backlog — each one a precise pointer to a clause the rubric never wrote — then ships the revision and rechecks that spread.
Say this in an interview: "My Monday analysis isn't a report, it's a to-do list. I sort items by disagreement, and the top five tell me exactly which guideline clauses to fix this week — the annotators were never the problem, the under-specified cases were."