Week 10 of 12 · The Lead

Who Needs Calibration?

Build the report that turns a pile of labels into a per-person coaching agenda — who's off, in which direction, and what you'll say

Day 48 ~70 minutes Build Day

Day 48 of 66

Today you build the diagnostic

Yesterday you learned to read an agreement table by hand. Today you write the code that produces it from raw labels, so that walking into a calibration session you already know who to talk to and about what. The full calibration_report.py is in the Try This box below (injected on build days). Type it in — don't paste — so every line passes through your hands.

What the starter script does

The version in the Try This box does the core move: for each item it finds the consensus label (the majority vote), then for each annotator computes their agreement-with-consensus rate and flags anyone below a threshold as "calibrate." That's the skeleton. The build today is extending it from a rate into an agenda — because a rate tells you who, and a lead needs why and what to say.

The build: rate → direction → coaching note

A bare agreement percentage is a leaderboard, and we spent yesterday establishing that a leaderboard is the wrong artifact. Extend the script so each flagged person gets a real agenda entry. Three additions, in order:

Core Theory — what to add

1 · Capture the direction of each disagreement

When an annotator's label ≠ consensus, record which way they went. For ordinal labels (reject < borderline < accept) that's "more lenient" or "more harsh" than consensus. Tally those per person. If the tally leans hard one way → systematic bias. If it's roughly balanced → scattered / random.

2 · Turn the pattern into a note, not a verdict

Map the pattern to a coaching line: systematic-harsh → "coach on what the guideline allows"; systematic-lenient → "coach on the reject bar"; scattered → "check workload / re-read guideline together." The note names the fix, never the failing.

3 · Attach the evidence items

For each flagged person, list the specific item IDs where they split from consensus. You never walk into a coaching conversation with a number — you walk in with three items to look at together. The item list is what makes the feedback specific (Day 46's first rule).

Build steps
  1. Run the Try This calibration_report.py as-is. Confirm it prints a per-annotator agreement rate and a flag.
  2. Add a direction tally: for ordinal labels, count how often each annotator was more-lenient vs more-harsh than consensus. Print the lean.
  3. Add a per-person item list of their disagreements, and a one-line coaching note derived from the pattern (systematic vs scattered).
  4. Add the honest guard: if agreement is low across everyone on the same items, print "GUIDELINE GAP — fix the doc, not the people" instead of flagging individuals.

Two things the code must not do

The script is a sharp tool, and sharp tools cut the wrong way easily. Two guards keep it honest — both straight from yesterday.

Don't auto-correct the expert; don't punish scatter

First: flag, don't auto-correct. A low-agreement domain expert gets the same flag as anyone, but the action is "investigate / adjudicate," not "drive to consensus." Bake that into the note for any annotator you've tagged as an expert — the code suggests adjudication, not correction. Second: scatter is not a verdict on the person. If a person's disagreements don't lean, the note must say "check conditions / guideline," not "low performer." The report's job is to start the right conversation, not to end it.

This is the difference between a tool that builds a team and one that quietly destroys it. The same numbers, framed as a ranking, push people to label defensively toward consensus and stop flagging the ambiguous cases that actually improve the guideline — the exact behavior you spent the whole track learning to value.

Why this matters at the operation level

Hand-reading agreement works for a 4-person team and one batch. It does not scale to a 30-person vendor team across a dozen task types — and that's the scale a Human Data Lead operates at. The report you build today is the thing you run every week before calibration: it reads the whole team's last batch and hands you the meeting agenda automatically. The judgment stays human; the bookkeeping becomes code.

The lead's loop, in one line

Labels in → agreement-with-consensus per person → direction → coaching agenda → calibration session → guideline edit → better labels next week. calibration_report.py is the first arrow. The session (Day 50) is the last. You now own the whole chain.

Your work today

Ship the agenda generator

From rate to agenda

  1. Build the three extensions above on top of the Try This script: direction tally, per-person disagreement items, and a pattern-derived coaching note.
  2. Add the guideline-gap guard so team-wide scatter blames the document, not a person.
  3. Run it on a believable multi-annotator set that includes one systematically-harsh annotator and one correct-but-low-agreement expert. Confirm the report coaches the first and flags the second for adjudication — different actions, same number.
  4. Save the output. It's the centerpiece of your Leadership Playbook artifact (Day 50 / Week 10 portfolio).
The lead move

An IC can hand-read a 4-person agreement table. A lead writes the code that turns 30 annotators' raw labels into a per-person coaching agenda — pattern, evidence items, and a note that names the fix, not the failing — because the judgment has to stay human while the bookkeeping scales. The same script, framed as a leaderboard, trains your team to label defensively and stop flagging the ambiguity that improves the guideline.

Say this in an interview: "My calibration report flags low agreement but never auto-corrects — a low-scoring expert routes to adjudication, and team-wide scatter prints 'fix the guideline, not the people.' I build the tool so it starts the right conversation instead of ending it."

Today's Takeaways