The same agreement numbers that diagnose a guideline can diagnose a person — and the difference between coaching and punishing lives in how you read them
Day 47 of 66
In Week 3 you learned agreement metrics as a verdict on the guideline: low kappa means the task is underspecified. This week you turn the same lens on the team. The headline number is per-annotator agreement with consensus — for each person, what fraction of their labels match the label the rest of the team converged on.
This single column is the most powerful and the most dangerous artifact you'll own. Powerful because it tells you exactly who to coach and on what. Dangerous because the obvious thing to do with it — rank the team and lean on the bottom — is almost always the wrong thing.
Per-annotator agreement is a diagnostic for a coaching agenda, not a leaderboard for punishment. A low number tells you that someone diverges; it does not tell you why, and the "why" — systematic bias, random error, or a correct expert disagreeing with a wrong consensus — determines whether you coach, clarify the guideline, or change your own mind.
Two annotators can both sit at 70% agreement and need completely opposite interventions. The rate alone hides which. You have to look at the direction and shape of the disagreements.
When someone disagrees, they disagree consistently in the same direction: they reject what the team accepts, or score one dimension a full point low every time. This is a fixable, coachable pattern — they've internalized one rule differently from the team. A short conversation anchored on three examples usually closes it: "you're treating shallow focus as an artifact; the guideline doesn't." Systematic = a recalibrated reading, not a bad annotator.
When someone's disagreements have no pattern — sometimes high, sometimes low, no consistent dimension — that's noise. Its causes are different: the person is rushing, distracted, fatigued, or the guideline genuinely doesn't cover these cases and everyone is guessing. Scattered error across the whole team is almost never a people problem — it's a guideline problem wearing a people-problem costume. Fix the document before you "fix" the person.
| Pattern | Likely cause | Right intervention |
|---|---|---|
| Consistently lenient / harsh in one direction | Internalized a rule differently (bias) | Coach on that specific rule, anchored on examples |
| One dimension always off, others fine | Misread that dimension's definition | Clarify that dimension; add a worked example |
| Scattered, no pattern — one person | Fatigue, rushing, distraction | Check workload/throughput pressure, not the rubric |
| Scattered across the whole team | Guideline gap — everyone's guessing | Rewrite the guideline; it's not a people issue |
Never act on the agreement rate alone. For any flagged annotator, pull their actual disagreements and ask one question: do these point one way, or do they scatter? One way → coach the bias. Scatter on one person → check their conditions. Scatter on everyone → fix the guideline. That branch is the entire skill.
Consensus is the majority's reading. The majority is not truth. When your one radiologist, colorist, or native-speaker reviewer sits at the bottom of the agreement column, the lazy move is to "correct" them toward the consensus — and you may be systematically training the model on the wrong answer, because the rest of the team simply can't see what the expert sees.
A domain expert who disagrees with consensus is a signal to investigate, not a number to auto-correct. The honest read might be: the expert is right and the consensus is wrong, which means the guideline (and the team's training) needs to absorb what the expert knows. A lead who blindly drives everyone to consensus optimizes for agreement and against accuracy — the two are not the same thing, and confusing them is how a confidently-wrong dataset gets built.
How you tell the difference: take the items where the expert split from consensus and adjudicate them on the merits — ideally with a second expert or a gold answer, not a vote. If the expert keeps being right, the fix is upstream: update the guideline, and consider whether expert items should carry more weight or route to expert-only consensus. If the expert is genuinely off, it's a normal coaching case like any other. Either way you investigated before you judged.
The output of this analysis is never a sorted list you wave at people. It's a per-person agenda: for each annotator who's off, what the pattern is, which items show it, and what the conversation or guideline edit will be. That's exactly what tomorrow's build produces in code.
An evaluator sees a low agreement number and assumes the annotator is wrong. A lead reads the direction of the disagreements first — one-way bias is coachable, scatter is fatigue or a guideline gap, and a low-scoring expert may be the only person who's right. The rate tells you that someone diverges; it never tells you why.
Say this in an interview: "I never rank a team on agreement-with-consensus and lean on the bottom. Optimizing for agreement isn't optimizing for accuracy — drive everyone to consensus and you'll happily train the model on whatever your experts can see and the majority can't."