Compute Cohen's kappa by hand once, read it correctly forever, and know which cousin to reach for
Day 12 of 66
Yesterday you saw that percent agreement is inflated by chance. Cohen's kappa fixes that with one tidy idea: measure how far the raters got from chance toward perfection.
κ = (Po − Pe) / (1 − Pe)
Po = observed agreement — the fraction of items the two raters labeled identically (that's just percent agreement).
Pe = expected-by-chance agreement — how often they'd agree if each rater kept their own label frequencies but assigned them at random.
Read the formula in plain words: the top is "how much they beat chance" (observed minus chance). The bottom is "how much they could have beaten chance" (perfect minus chance). So κ is the share of the available, non-luck agreement they actually captured. κ = 1 is perfect; κ = 0 is no better than chance; κ < 0 is worse than chance (systematic disagreement).
Two raters label 20 images as accept or reject. Cross-tabulate their decisions into a 2×2 table — each cell is a count of items.
| B: accept | B: reject | A total | |
|---|---|---|---|
| A: accept | 10 | 2 | 12 |
| A: reject | 3 | 5 | 8 |
| B total | 13 | 7 | 20 |
Percent agreement said 75% — sounds fine. Kappa said 0.47 — only "moderate." The 22 points of difference were luck: with these label frequencies, raters would agree 53% of the time by accident. Kappa stripped the luck out. Always report kappa, not raw percent, when the labels are unbalanced.
The standard interpretation bands come from Landis & Koch. Memorize the shape; keep the table handy.
| κ range | Interpretation |
|---|---|
| < 0.00 | Poor (worse than chance) |
| 0.00 – 0.20 | Slight |
| 0.21 – 0.40 | Fair |
| 0.41 – 0.60 | Moderate |
| 0.61 – 0.80 | Substantial |
| 0.81 – 1.00 | Almost perfect |
κ < 0.4 → fix the guidelines, not the people. The task isn't decided for the raters; that's a specification problem you own. κ > 0.6 → solid, the task is well-posed. Between 0.4 and 0.6, look at which items disagree before you act. Our worked example (0.47) is a "look closer" task.
One caveat: "good" depends on task subjectivity. A crisp factual label should clear 0.8 easily; a genuinely subjective aesthetic judgment may top out around 0.5 even with a great guideline — and that's information too. The number is a thermometer, not a verdict; your job is to read what it's telling you about the task.
The de Bruijn article shows Fleiss in worked code; the Fleiss reference has the formula and the interpretation table. For why this matters to training and not just to your spreadsheet, skim the data-quality discussion in the RLHF Book.
Kappa measures agreement, and agreement is not correctness. A team can post κ = 0.9 and all be wrong together. Kappa tells you the task is well-specified and your raters are consistent — it tells you nothing about whether the labels are right. For that you need a gold set, which is Day 14.
An IC quotes the kappa. A lead reads what the number is telling them about the task — a low kappa is a guideline problem to own, not a people problem to discipline — and picks the right instrument (Cohen, Fleiss, Krippendorff) for the data they actually have.
Say this in an interview: "I treat kappa as a thermometer for the task, not a verdict on the raters. Below 0.4 I fix the guideline before I touch the people, and I match the metric to the data — Fleiss for multi-rater, Krippendorff for ordinal scales."