Put the multimodal rubric to work at scale: score a batch, auto-flag the borderline ones, and measure the aesthetics agreement problem in real numbers
Day 23 of 66
You can describe the dimensions perfectly. Today you make them operate on a whole batch — because a lead never scores one image, they run a pipeline over thousands and decide which handful need a human's eye. The same Day 21 rubric, applied programmatically, turns "I have a good eye" into "I have a triage system." That's the altitude jump again.
The "Try This — On Real Data" box at the bottom of the page contains the full image_eval.py. Type it in rather than paste — every line is teaching something. It scores a batch against weighted dimensions, auto-flags the borderline items, and sets up the aesthetics-agreement measurement we work through here.
Three jobs, in order — each is a real lead responsibility:
Each image gets a per-dimension score; the script computes the weighted final exactly as your Week 2 rubric.py did — adherence weighted highest, aesthetics middle, artifacts, text small. Same weights, now running over a list instead of one sample.
The clear accepts and clear rejects can route automatically. The borderline band (say 2.5–3.5) is where machine judgment is least trustworthy, so those get flagged FLAG for human review. This is triage: spend scarce human attention only where the score is genuinely uncertain. A lead designs where that band sits.
The script ends by reminding you that aesthetics is the lowest-agreement dimension — and pointing at the decision: average raters, or define it tighter. That's not a code problem; it's the judgment call you make next.
This is the day those two weeks meet on real data. You measured agreement in Week 3 (percent agreement, Cohen's kappa). You learned to tighten definitions with anchored examples in Week 2. Aesthetics is exactly the dimension that forces you to use both.
So extend the build: have two raters (you, plus a colleague — or you twice, a day apart) score the aesthetics dimension on the same 10 images. Then compute their agreement using the Week 3 logic.
| Aesthetics agreement you measure | What it's telling you | The lead's move |
|---|---|---|
| High (kappa > 0.6) | Your definition is already anchored enough | Keep it; single-rater is fine |
| Low (kappa < 0.4) | Raters are scoring on personal taste | Tighten with anchored examples, OR average 3+ raters |
| Low even after anchoring | Residual genuine subjectivity | Average raters and report the spread, not a single number |
The number isn't the point — the decision the number triggers is. A tutor reports "aesthetics agreement is 0.31." A lead says: "aesthetics agreement is 0.31, so I'm anchoring the definition with three exemplar images and re-measuring; if it's still under 0.5 we move to 3-rater averaging." That sentence is the job.
Run image_eval.py from the Try This box on a sample batch first to see the flagging work. Then swap in real generations you've evaluated (policy-safe) and your actual rubric weights. Watch which images land in the FLAG band — those are precisely the ones a real pipeline would route to you.
Three things to capture. (1) Which items got auto-flagged, and do you agree they were the genuinely ambiguous ones? (2) Your measured aesthetics agreement number. (3) Your written decision — tighten, average, or both — with the threshold that would change your mind. Screenshot the output; this is your Week 5 build artifact and it feeds Day 24's preference pairs.
A labeler reports the aesthetics-agreement number. A lead turns the number into the decision it triggers: a batch scorer that auto-flags only the genuinely borderline band, and a measured kappa that dictates whether to anchor the definition or average more raters — with the threshold that would change the call stated up front.
Say this in an interview: "When aesthetics agreement comes back at 0.31, I don't just report it — I anchor the definition with exemplar images and re-measure, and if it's still under 0.5 we move to three-rater averaging. The number is only useful because of the decision it forces."