The dimensions that decide whether a generated image is good — and which one is the trap that breaks rater agreement
Day 21 of 66
Most people who become a Human Data Lead come up through text. You didn't. You came up rating Grok images and correcting video descriptions — which means this week isn't new material to learn so much as tacit skill to make explicit. The judgment you already make by eye, you'll now name, weight, and defend. That naming is the whole difference between a tutor who scores well and a lead who can teach a hundred people to score the same way.
"This image is good" is not a label — it's a feeling. A lead's job is to decompose that feeling into dimensions a stranger could score the same way you do. For generated images, the field has converged on a small, stable set. Learn them as a vocabulary, not a checklist.
When you look at a Grok image and decide it's good or bad, you are unconsciously weighing these. Drag them into the light:
Does the image actually contain what was asked for? "A red bicycle leaning on a blue door" — is the bicycle red, is the door blue, is it leaning? This is the primary dimension because a gorgeous image of the wrong thing is a failed generation. It's also the most objective: two raters with the prompt in front of them usually agree. Adherence is checkable against the request, so agreement is high.
Composition, lighting, color, "appeal." This is the dimension everyone wants to lead with and it is the single lowest-agreement dimension you will ever manage. Taste varies; what reads as moody to one rater reads as muddy to another. Hold this thought — it's the center of gravity for the whole week.
Mangled hands, six-fingered grips, warped text on signs, melted backgrounds, anatomy that breaks, objects that fuse. These are defects, and like adherence they're fairly objective: a sixth finger is a sixth finger. Your trained eye catches these faster than a text-native lead ever will.
If the prompt asks for legible text in the image ("a sign that says OPEN"), can you read it, is it spelled right, is it placed sensibly? Generation models historically fail here, so it earns its own axis rather than hiding inside "technical."
Policy compliance. This isn't averaged in with the others — it overrides them. A flawless image that violates policy is a reject, full stop. Treat safety as a gate that can zero out everything else, exactly as your Week 2 rubric encoded it.
Adherence and artifacts are verifiable against an external reference (the prompt, reality) — so raters converge. Aesthetics is internal to the rater — so raters scatter. A dimension's inter-annotator agreement is mostly a function of how external its reference is. Remember that sentence; it's how you'll predict which dimensions will fight you before you ever collect a label.
Here is the collision you'll spend the next four days resolving. Aesthetics matters to users, so you can't drop it. But it has the lowest inter-annotator agreement of any dimension — meaning if you just ask raters "rate the aesthetics 1–5," you get noise dressed up as data. A lead facing this has exactly two honest moves:
What you cannot do is leave it ambiguous and pretend the resulting scores are clean. Either anchor the definition or average the raters. Most strong operations do some of both: anchor enough to kill the wild disagreement, then average to absorb the residual taste. Day 23 makes you measure this collision in code, on real data.
Look back at your Week 2 rubric. It had four weighted dimensions and a safety gate — prompt adherence weighted highest, aesthetics in the middle, technical/artifacts, and a small text slice. That was multimodal on purpose. You didn't write a generic text rubric; you wrote an image-generation rubric, because that's your edge. This week you'll extend the exact same structure to video (Day 22), run it on a batch (Day 23), and turn it into preference pairs (Day 24).
The principles behind a safety gate generalize cleanly from text to pixels — the Constitutional AI idea of "principles as a guideline document" is exactly how you'd specify what your safety gate rejects in a generated image. And the preference logic you'll meet again on Day 24 is the same chosen/rejected machinery from the RLHF Book — it doesn't care whether the output is a sentence or a frame.
Keep this — it's the seed of the finalized multimodal rubric you complete on Day 25.
A tutor scores an image and moves on. A lead decomposes "this is good" into named dimensions — and knows before collecting a single label which ones will fight them, because a dimension's agreement is a function of how external its reference is. Adherence converges; aesthetics scatters; predicting that is the lead's edge.
Say this in an interview: "I can tell you which dimensions of an image rubric will have low rater agreement before we collect anything — the more internal a dimension's reference, the more it scatters, and aesthetics is the worst offender. I plan the collection around that."