Week 2 of 12 · The Craft

Ship the Rubric

Synthesize the week and finish the test that matters: could a stranger apply your rubric with no explanation from you?

Day 10 ~60 minutes Review

Day 10 of 66

What you now know

In one week you went from following rubrics to authoring one — the highest-leverage artifact a data lead owns. Say the spine back without looking:

The Week 2 spine

Self-quiz (no notes)

Answer out loud or in writing. If any answer is shaky, reread that day before starting Week 3.

  1. Your team's labels are inconsistent. A teammate says "let's add a QA review layer." Give the data-centric counter-move and say why it's cheaper.
  2. Why must the safety dimension gate rather than be weighted into the average? Give the failure mode if you weight it.
  3. A rater asks "what's the difference between a 3 and a 4 on aesthetics?" What's missing from your rubric, and what's the fix?
  4. You're scoring real outputs and the rubric's verdict disagrees with your gut. What are the only two possibilities, and what do you do about each?
  5. What is a guideline changelog, and name two reasons a lead keeps one.
Answer key (peek after you try)

1) Fix the guideline, not the symptom — find the cases they split on and remove the ambiguity. It's cheaper because inspection costs scale linearly with volume forever, while a guideline fix is paid once and amortized across all future items. 2) Because a weighted safety score lets a high-quality item average past a violation (5/5/5 with safety 1 still lands ~4.6 and gets accepted); a gate auto-fails the whole item regardless of other scores. 3) The scale point isn't anchored — add an observable description of what a 3 vs a 4 looks like, ideally with a calibration example at the boundary. 4) Either the rubric is underspecified (fix it) or your gut leans on an unstated factor (name it as a dimension or admit it shouldn't count) — never silently override; investigate. 5) A versioned record of what changed in the guideline and why; reasons include: it makes quality drops auditable ("what changed?"), it's reversible, and it proves you iterate on evidence not opinion.

The test that defines "shipped"

A rubric isn't done when you can apply it. It's done when someone else can apply it correctly with zero explanation from you. That is the bar for shipping, and it's the bar your portfolio artifact must clear.

The reusability standard — memorize it
  1. Hand it over cold. Could a new annotator read only the document — no conversation with you — and start scoring? If they'd need to ask you a question, the answer to that question belongs in the rubric.
  2. The boundaries are anchored. Every place two people might split has a calibration example or a counter-example sitting on it.
  3. The unacceptable is gated. Safety and any non-negotiable failure auto-fail, not average in.
  4. The unknown has a home. There's an explicit skip/uncertain/flag path for cases the guideline doesn't yet resolve — and a changelog showing it's a living document.
Why this is the whole leadership move

A rubric only you can apply doesn't scale past one person — it isn't leadership, it's just your own good taste written down. A rubric a stranger can apply scales to ten annotators, then a hundred, with consistent output. That transferability is the entire reason rubrics are the highest-leverage thing a lead does: you encode your judgment once, and a whole team produces it. Build for the stranger, not for yourself.

Close the week

Wrap-up

Before Week 3

Up next — Week 3: Measuring Data Quality

You'll stop saying "I think the rubric is clear" and start proving it. Week 3 gives you the numbers — inter-annotator agreement, Cohen's kappa, gold sets, and sampling math — so you can say "agreement on this task is 0.41, that's a guideline problem" and back it with data. The disagreements you logged this week become the inputs you'll measure.

The lead move

A great annotator's judgment lives in their head and ships with them when they leave. A lead's judgment ships in a document a stranger can apply correctly with zero explanation — which is the only version that scales from one person to ten to a hundred at consistent quality. The altitude jump is from "I have good taste" to "I encoded my taste once, and now a whole team produces it without me in the room."

Say this in an interview: "My bar for 'shipped' isn't that I can apply the rubric — it's that a new annotator can read only the document, no conversation with me, and reach my answers. If they'd have to ask me a question, that answer belonged in the rubric; I build for the stranger, because that transferability is the whole reason a guideline is the highest-leverage thing a lead owns."

Week 2 Takeaways