Week 3 of 12 · The Craft

Compute Real Agreement

Turn yesterday's hand-calc into running code — then use the disagreements as a map of your guideline's blind spots

Day 13 ~75 minutes Build Day

Day 13 of 66

From formula to function

You computed Cohen's kappa by hand yesterday. Today you make a machine do it — once — so you can run it on real label data any time someone asks "is this task well-specified?" Building the metric yourself, rather than importing a black box, is the point: when you've watched the chance term move the answer, you'll never misread a kappa again.

The code is in the Try This box below

The "Try This — On Real Data" box (injected for build days) contains agreement.py. Type it in rather than pasting — read every line. It computes percent agreement, the chance term, and kappa exactly as you did by hand, then lists the items where the two raters disagreed.

Reading your own number

When it runs, you'll get three numbers and a list. Read them in order, the way a lead would in a quality review:

Observed agreement — the headline most people stop at. Note it, then distrust it.
Expected by chance — the gap between this and observed is the luck you're about to strip out. The more lopsided your label distribution, the bigger this term, the more kappa will correct.
Cohen's kappa — the honest number. Drop it on yesterday's bands: <0.4 means the guideline needs work, >0.6 means the task is solid. Don't reach for "retrain the annotators" until you've ruled out "the task was underspecified."
The disagreement list — the most valuable output on the screen. Read on.

Disagreements are a precise map of guideline gaps

This is the Week 2 lesson made quantitative. Every item where two careful raters split is a place your guideline failed to decide the case for them. The disagreement list isn't noise to average away — it's a ranked to-do list for your next guideline revision. Pull each disagreed item, ask "what rule would have forced the same answer?", and write that rule. That loop — measure agreement → inspect disagreements → tighten the guideline → re-measure — is the highest-leverage move you own as a data lead.

What a low kappa is — and isn't — telling you

Diagnose before you prescribe

Low κ, disagreements cluster on one label or one item type → a specific guideline hole. Fixable with one rule + a worked example.
Low κ, disagreements scattered everywhere → the task may be genuinely subjective, or the scale too fine. Consider collapsing categories or accepting a lower ceiling for this dimension.
Low κ on one rater vs everyone else → that's a calibration issue, not a guideline issue (Week 10) — but you can only see it when you have 3+ raters and compute pairwise.
High κ → consistent. Reminder from Day 11: this does not mean correct. Go check the gold set (tomorrow).

Run it on your real work

The sample data proves the math. The value is in your data. Pull two raters' labels on the same batch from a real xAI task — even 15–20 shared items is enough — swap them into agreement.py, and run it. Then open the disagreement list and write one guideline sentence for each split. That artifact — a kappa plus a list of guideline fixes derived from it — is exactly what makes a quality review credible. Screenshot the output; it goes in your portfolio next to your Week 2 rubric.

Keep it policy-safe: use labels and item IDs, not the raw flagged content, when the task touches sensitive material. You're measuring the labeling process, and you can do that with the labels alone.

The lead move

An IC runs the agreement script and reports the kappa. A lead treats the disagreement list as the real output — a ranked map of where the guideline failed to decide the case — and owns the loop that turns each split into a new rule, then proves agreement climbed.

Say this in an interview: "The kappa is the headline, but the deliverable is the disagreement list. Every item two careful raters split on is a precise pointer to a guideline hole, so I write a rule for each and re-measure to prove the fix moved the number."

Today's Takeaways

You have running code that reproduces your hand-calc: observed, chance, and kappa.
Read the gap between observed and chance — that's the luck kappa removes.
The disagreement list is the deliverable: a ranked map of where your guideline failed to decide the case.
Low κ is usually a guideline problem; the fix loop is measure → inspect → tighten → re-measure.