The code is easy — the prompts, the controls, and the pass bar are where evals earn trust
Day 19 of 60
Day 18's code is twenty lines. The hard part of evaluation was never the arithmetic — it's the set the arithmetic runs over. A two-sided scorecard computed on a biased, contaminated, or unrepresentative set produces a number that's precise and wrong. Today you design the eval set properly: the prompt classes, the controls that keep it honest, the metrics, and the pass bar you'd defend to a release committee.
An eval set is a sampling claim: "scoring well on these prompts predicts safe behavior in deployment." Every design decision either strengthens or quietly breaks that claim. The flattering-but-broken eval — green checkmark, dangerous model — almost always fails at the set, not the code.
A harmful class and a benign class — because the scorecard has two axes. The benign class must include benign look-alikes (surface features of harm, fully legitimate intent), or your over-refusal number is measured on softballs and reads better than reality.
Span your harm taxonomy (Week 2) across categories and severities, and span benign requests across the domains real users actually bring. A set that's all one category measures one thing and implies everything.
Could these prompts be in the model's training data? Use freshly authored or held-out items, paraphrase known benchmarks, and keep a private split you never publish. A leaked set measures memorization and silently inflates the score.
Report safe-refusal, harmful-compliance, and over-refusal — and justify each. Then set the bar before you see results (e.g. "harmful-compliance < 2%, over-refusal < 10%"). A pass bar chosen after the fact isn't a bar; it's a rationalization.
If you decide what counts as "passing" after seeing the numbers, you'll unconsciously fit the bar to the result you want. Pre-registering the bar — and the metrics, and why each matters — is what makes the eval a decision tool instead of a justification. This is the same discipline that keeps red-team rankings honest in Week 3.
The dangerous eval isn't the one that fails — it's the one that passes a model it shouldn't. Learn to smell it. Warning signs: a suspiciously high safe-refusal rate with an over-refusal axis that's missing or measured on trivial prompts; benign prompts with no real look-alikes; a public set the model likely trained on; a pass bar that appeared after the results. Your job designing an eval is to try to break your own set before it breaks a release decision.
For your draft eval set, write down the single way it could pass a genuinely unsafe model. That sentence is your eval's weakest link — and naming it is the difference between an eval you trust and one you merely hope is fine. (You'll name this weakest link again tomorrow in the Week 4 review.)
If your design scales scoring with an LLM-as-judge (Day 18), the design doc must name the biases you'll control for: position/verbosity preference, self-preference for the judge's own style, leniency on polite refusals, and sensitivity to formatting. The control is calibration against human labels plus reporting the human-judge disagreement rate. An eval doc that uses a judge but doesn't account for its bias is hiding a variable in plain sight.
A novice grades their eval by whether it runs. An expert grades it by whether it could pass a model that doesn't deserve to pass — and pre-registers metrics and the pass bar before seeing a single result, so the eval is a decision tool, not a rationalization. The altitude jump is from "I scored the model" to "I can defend that this score predicts deployment safety, and here's the one way it might not."
Say this in an interview: "I treat an eval set as a sampling claim and attack it before I trust it: balanced harmful and benign classes with real look-alikes, contamination controls, metrics and a pass bar I pre-register before seeing results. The question I'm really answering isn't 'did the model pass' — it's 'could this set pass a model that shouldn't,' and I write that weakest link down."