Week 3 of 12 · The Craft

Diagnose a Label Set

Lock in the week, prove it with a self-quiz, and write the one-paragraph quality diagnosis that turns numbers into a decision

Day 15 ~60 minutes Review

Day 15 of 66

What you now know

This was the most technical week of Phase 1, and it's the one that earns you authority. You can now answer "is our data good?" with numbers and a next step. Say the spine back without looking:

The Week 3 spine

Self-quiz (no notes)

Answer out loud or in writing. If any answer is shaky, reread that day before Week 4.

  1. Two raters show 90% percent agreement but κ = 0.15. What's going on, and which number do you report?
  2. Your task scores a beautiful κ = 0.88. A teammate says "great, the data's high quality." What's wrong with that conclusion, and what would you check next?
  3. What's the difference between a gold label and a consensus label — and when do you use each?
  4. You audit 100 random labels and find 91 correct. How do you report that to leadership? What would you need to do to tighten the estimate to ±3%?
  5. Your image-quality task sits at κ = 0.38. Is this a people problem or a guideline problem, and what's your first move?
Answer key (peek after you try)

1) The labels are lopsided, so chance agreement is high and percent agreement is inflated — report kappa (0.15 = only "slight," near-chance). 2) κ measures consistency, not correctness; they could be consistently wrong ("Agree + Wrong"). Check accuracy against a gold set. 3) Gold = documented correct answer (ground truth, for measuring accuracy); consensus = majority vote (for resolving disagreements at scale). Consensus can be confidently wrong. 4) "≈91% accurate, n = 100, roughly ±5 points." To reach ±3% you need a larger sample — error shrinks with √n, so roughly ~3× more items. 5) κ < 0.4 → guideline problem first. Inspect the disagreement items, find the undecided cases, write rules + worked examples, then re-measure — don't start by blaming raters.

The deliverable: a one-paragraph quality diagnosis

This is the artifact that proves the week. Take a real (or realistic) label set and write a single tight paragraph that uses every tool from the week and ends in a recommended fix. This is the exact shape of what you'd post in a weekly quality review.

Write the diagnosis

One paragraph, four moves

  1. Agreement: state the kappa and what it means. "Cohen's κ = 0.41 across two raters — moderate."
  2. Accuracy: state gold-set / sample accuracy with its error bar. "Gold accuracy 86%, n = 120, ≈±6 points."
  3. Diagnosis: name the dominant problem from the evidence. "Disagreements cluster on borderline aesthetic cases — a guideline gap, not a people problem."
  4. Recommended fix + re-measure: "Add a tie-breaker rule + 3 worked aesthetic examples, then re-run agreement on the next batch to confirm κ rises above 0.6."

Aim for 4–6 sentences. If you can write this paragraph from real numbers, you can run a quality review — and that's the line between doing labels and diagnosing a labeling operation.

Up next — Week 4: Preference Data & RLHF

You've measured the quality of the data. Now you'll see where it goes: how the pairwise comparisons you make become a reward model, and how you'd design and scale preference collection. You'll build a real reward-model-ready dataset in the JSONL shape a training team ingests — your next portfolio artifact.

The lead move

An IC produces the labels. A lead produces the quality diagnosis — agreement, accuracy, named problem, recommended fix, re-measure — in a single tight paragraph that turns a pile of numbers into a decision a stakeholder can act on.

Say this in an interview: "I can take any label set and write the one-paragraph quality review: kappa and what it means, gold accuracy with its error bar, the dominant failure mode, and the fix to re-measure against. That paragraph is the difference between doing labels and running a labeling operation."

Week 3 Takeaways