Define a scale so precisely that two people apply it identically — then weight what matters and gate what's unacceptable
Day 7 of 66
Yesterday you learned why the rubric is the job. Today you learn the core mechanical skill: defining a scale so tightly that two reasonable people, scoring the same item, land on the same number. That property has a name — high inter-annotator agreement — and almost everything in this lesson exists to produce it.
A great rubric removes interpretation latitude. Every place your guideline lets an annotator decide for themselves what a word means is a place two of them will decide differently — and that difference becomes noise in your training data. You are not writing to inspire judgment; you are writing to replace it with a shared, explicit standard.
The most common rubric failure is a scale whose points are named but not defined. "Rate aesthetic quality 1–5" is useless: your 4 is my 3. The fix is anchoring — attaching a concrete, observable description (and ideally an example) to every single point.
An anchored point answers the question "what exactly does a 3 look like?" with something an annotator can check against reality, not a feeling. Compare:
The anchored version names observable features. Two annotators checking the same image against it will converge, because they're matching the picture to a description, not consulting their personal taste. Anchor the top, the bottom, and every step between — the middle points are where disagreement concentrates, so they need the most precise anchors.
Before you anchor points, decide how many points there should be. Using the wrong scale shape guarantees low agreement no matter how well you write.
| Scale | Shape | Use it when… | Watch out for |
|---|---|---|---|
| Binary | Yes / No, Pass / Fail | The question has a genuine right answer (policy compliance, "does it contain text?") | Forcing a fuzzy quality call into two buckets — you'll lose real signal |
| Ordinal | Ranked categories (reject / borderline / accept) | There's an order but the gaps aren't equal; you want clear action tiers | Don't average ordinals as if they were numbers — the gap from reject→borderline isn't the same as borderline→accept |
| Likert | 1–5 (or 1–7) numeric | You need a graded quality measure with enough resolution to compare and average | Central-tendency bias (everyone clumps at 3); unanchored points; too many points (1–10 rarely beats 1–5) |
If the question is "is this allowed?" use binary. If it's "how should we route this?" use a small ordinal set with named tiers. If it's "how good, on a continuum we'll average and compare?" use an anchored 5-point Likert. More points feel more precise but usually just spread the disagreement — 5 well-anchored points beat 10 vague ones every time.
Most real quality judgments are multi-dimensional. An image has prompt-adherence, aesthetics, technical execution, and so on. The naive move is to treat them equally. That's almost always wrong — it lets a gorgeous image that ignores the prompt score the same as a plain image that nails it.
Instead, assign each dimension a weight that encodes how much it matters for this task's purpose. For image generation, prompt-adherence usually dominates — a beautiful picture of the wrong thing is a failure. Make the weights explicit numbers that sum to 1, write down why each one is what it is, and the rubric now carries your priorities, not each annotator's.
When you set prompt-adherence to 0.45 and aesthetics to 0.25, you are telling the entire operation — and ultimately the model — that following the prompt matters nearly twice as much as looking good. Weights aren't a math detail; they're how a lead encodes priorities into thousands of downstream labels. Choose them deliberately and defend them.
Some failures can't be averaged away. A weighted score quietly assumes every dimension is tradeable — that enough beauty can offset a little harm. For safety, that assumption is unacceptable. The fix is a gating dimension.
A safety gate is a dimension that auto-fails the entire item when it trips, regardless of how high every other score is. It sits outside the weighted average: first you check the gate; only if it passes do you compute the quality score. A flawless, stunning image that violates policy scores zero — not a high number with a small penalty. The gate makes "this is never acceptable" a hard rule instead of a soft weight.
The classic beginner error is to give safety a weight (say 0.10) and average it in. Then a 5/5/5 image with a safety score of 1 still lands at ~4.6 and gets accepted. Catastrophic. Safety, and any genuinely non-negotiable dimension, must gate, not contribute. If it trips, the item fails — full stop. You'll implement exactly this branch in code on Day 8.
An anchored, weighted, gated rubric is still just text until you attach calibration examples: a handful of real items, each with its correct score on every dimension and the one-line reasoning. These are the worked examples and counter-examples from Day 6, doing their real job — they're how a new annotator calibrates to your standard before touching live data, and how you settle a dispute ("we score it like example 4, here's why").
The best calibration examples sit right at the boundaries: the item that's just barely an accept, the one that's just barely a reject, the near-tie. Those are where latitude hides, so those are where your examples earn their keep.
Keep everything policy-safe and general — describe the task and standards, never paste confidential xAI content.
An annotator applies the scale they're handed and lives with its blind spots. A lead designs the scale's shape — choosing binary, ordinal, or Likert, anchoring every point, and deciding that safety gates instead of averaging in — so that the team's priorities are baked into the numbers, not left to each rater's taste. The altitude jump is from "I score what matters" to "I encode what matters into the instrument everyone scores with."
Say this in an interview: "Weights are a strategy statement, not a math detail — when I set prompt-adherence above aesthetics I'm telling thousands of downstream labels that the right subject beats a pretty wrong one. And I never let safety be a weight; it gates, so a stunning policy violation fails outright instead of averaging past."