Apply your rubric to real work — every hesitation locates an ambiguity it failed to resolve — then iterate to v2 and log why
Day 9 of 66
A rubric written at a desk is a hypothesis. Today you test it the only way that counts — by applying it to real samples during an actual shift. Reality always finds the gaps a quiet afternoon of writing never will, and that's exactly what you want it to do.
While you apply the rubric, pay attention to your own friction. The moment you hesitate — "is this a 3 or a 4?" — or the moment you'd score an item differently than a teammate would, you have located a precise ambiguity the guideline failed to resolve. Don't push past the hesitation. Mark it. Hesitation is not a personal failing; it's a free, exact pointer at the next sentence you need to rewrite.
This is the Judgment Ritual from Week 1 turned outward onto your own artifact: check the guideline, check the calibration examples, and where it's still unclear, flag and write down the ambiguity rather than guessing silently. The difference now is that you own the guideline, so you also own the fix.
The point of testing isn't to confirm your rubric works. It's to manufacture disagreements and hesitations on purpose, because each one is a defect you can fix before the rubric ever reaches a team. A rubric that produces zero hesitation on ten varied real samples is either perfect (it isn't) or your samples were too easy. Deliberately feed it the hard, weird, boundary cases — that's where guidelines earn their keep.
rubric.py). For every hesitation, write a one-line note: "Item X unclear because the guideline doesn't say Y."You wrote it, so you understand it — which makes you the worst possible judge of whether it's clear. The decisive test is whether someone else can apply it and reach your answers.
Hand your rubric and a few of the same samples to a colleague (or score a set, then have them score it cold). Compare. Every item where you disagree is a guideline gap you literally could not have seen alone — because in your head the missing rule was always "obvious." Their disagreement is the most honest agreement signal you'll get this week, and it previews the formal inter-annotator agreement metrics you'll compute in Week 3.
When you change a rubric, you must record what changed and why. This discipline — the guideline changelog — is one of the habits that separates a lead from a labeler, and it's a required part of your Week 2 portfolio.
A changelog makes your rubric auditable and reversible. When agreement drops next month, the first question is "what changed in the guideline?" — and a changelog answers it in seconds instead of a forensic investigation. It also proves the thing employers most want to see: that you improve specifications with evidence, not opinion. Eugene Yan's iteration section is the model — guidelines are living documents, and the changelog is their version history.
Everything you produce this week is a generic rubric for a task type — image-generation quality, video consistency, and the like. Do not paste confidential xAI prompts, internal guidelines, model outputs you can't share, or any proprietary data into your portfolio artifact. Describe the task and the standard in your own general terms. The skill is transferable and demonstrable without exposing anything sensitive.
An annotator pushes past a hesitation to get the item scored. A lead treats every hesitation as a located defect in the guideline — marks it, clusters the recurring ones, and ships a versioned fix with the evidence that it resolved the disagreement. The altitude jump is from "I score through the ambiguity" to "I own the loop that manufactures disagreements on purpose, then writes them out of the spec for good."
Say this in an interview: "I don't trust a rubric I can apply — I'm the worst judge of my own clarity, so I run a blind-apply test and let a colleague's disagreement find the gap I literally couldn't see. Then every change goes in a changelog with the problem, the fix, and the evidence, so when agreement drops next month the first question answers itself."