For years a lead calibrated you. This week you design the calibration — and "Lead" stops being a title and starts being a job
Day 46 of 66
Everything in Phase 1 and the first half of Phase 2 was about your judgment: a rubric you can run, agreement you can measure, a pipeline you can design. This week the unit of work becomes other people's judgment. A team of annotators — some faster than you, some who know a slice of the domain better than you ever will — produce labels, and the quality of that label set is now your responsibility, not theirs.
You have been the person being calibrated for years. You sat in the session where someone showed you the items you got "wrong" and walked the team toward a shared reading of the guideline. Today you learn to run that session — and then to onboard the people who'll sit in it, and to give the feedback that makes their next label better.
A calibration session, an onboarding checklist, and a feedback conversation are not "soft skills" bolted onto the technical work. They are the mechanism by which a guideline actually reaches a label. A perfect rubric that the team interprets five different ways produces five different datasets. Leading the team is guideline iteration, applied to humans instead of a document.
The word gets thrown around loosely. Here is the precise mechanism, because if you run it wrong it becomes a meeting where you tell people they're wrong — and that fixes nothing.
That last step is the one new leads skip, and it's the whole point. If the team converges in the room but the guideline still doesn't cover the case, you'll have the identical disagreement next month with the next batch — and any new hire will diverge again. Calibration that doesn't change the document is calibration you have to repeat forever.
Eugene Yan's guide to writing labeling guidelines ends on iteration — guidelines improve by being tested against real disagreements. A calibration session is exactly that iteration loop, run with the people instead of alone at your desk. Reread his iteration section with the team in mind: every disagreement you surface is a precise pointer to the next guideline edit. Calibration and guideline-writing are the same activity.
Calibration is not a thing you do when quality looks bad. By the time quality looks bad, you've already shipped a batch of drifted data. Run it on a fixed cadence — weekly is common while a task is new or a guideline is changing, dropping to biweekly once agreement is stable.
Onboarding is calibration's twin. A new annotator who starts labeling live data before they're calibrated is just injecting noise at full volume. The detailed onboarding checklist is your Field Day on Day 49 — today, hold the one principle:
No new annotator touches the real queue until they've labeled a gold set and a calibration set and cleared a quality bar. The gate is non-negotiable, and it applies to the senior domain expert too. Their domain knowledge is real; their reading of your guideline still has to be calibrated to the team's.
The domain expert is the delicate case. You may be onboarding a radiologist to label medical-image generations, or a colorist to judge cinematic grading, when you can't do their job. Two things are true at once: they will sometimes be right when they disagree with the consensus (the rest of the team simply lacks their eye), and they still have to interpret the same guideline as everyone else. The lead's posture is humble but not deferential: investigate the disagreement, learn from it, and decide whether the guideline should bend to the expert or the expert to the guideline. We make that fairness call rigorous tomorrow.
You will have to tell people their labels are off. Do it badly and you get demoralized annotators who label defensively or churn out; do it well and you get a team that self-corrects. The difference is entirely in how the feedback is framed.
| Crushing feedback | Guideline-anchored feedback |
|---|---|
| "You're being too harsh on these." | "On items 4 and 9 you rejected for soft focus; the guideline (§3) treats intentional shallow depth-of-field as acceptable. Here's the example." |
| "Your accuracy is the lowest on the team." | "Your scores skew one direction on aesthetics — let's look at three together and find where our readings split." |
| "Just match the consensus." | "You disagreed with consensus on these five — walk me through your reasoning; if the guideline's unclear, that's a fix we both win from." |
Specific — point at items, not the person ("on items 4, 9," never "you always"). Guideline-anchored — the standard is the document, not your taste, which makes it impersonal and contestable. Two-way — ask for their reasoning; half the time the "error" is a guideline gap and they're the one who found it. Feedback that flows only downward turns annotators into machines that guess what you want.
A tutor sits in the calibration session and gets corrected. A lead runs it — and knows the real output isn't a calibrated team, it's a guideline edit. Convergence that never touches the document is convergence you have to repeat with every batch and every new hire.
Say this in an interview: "I run calibration on a cadence, and I judge it by one output — did the guideline change? If the team agrees in the room but the doc still doesn't cover the case, we'll re-diverge next month, so the meeting failed."