Build the eval harness that catches its own judge in the act — order-swap control, self-consistency, and a win rate you can defend
Day 28 of 66
Yesterday you learned the three biases. Today you build the harness that neutralizes the most concrete one — position bias — and turns "the judge might be biased" into "here is exactly how many of its verdicts I can trust." The "Try This" box below contains llm_judge.py. Type it in line by line; the whole lesson is in the four lines that handle the swap.
The judge() function in the Try This box below is a stub that just prefers the longer answer — which is a real bias, deliberately. You're not wiring a model today; you're building the control logic around any judge. Once the swap-and-consistency machinery is right, swapping the stub for a real model call (the honors task) is a one-line change.
Every line in the code maps to one idea from Day 27. Read the code as the implementation of these four moves.
Judge A-then-B, then judge B-then-A. A judge with no position bias will name the same answer as the winner both times — its pick flips slots but not substance. This double-run is the entire mechanism for exposing position bias.
If the judge picks A in the first run and still picks the same answer after the swap, the verdict is consistent — it survived. If it flips to favor whatever is now in the preferred slot, the verdict was driven by position, not quality, and you discard it. You only ever score the comparisons the judge agreed with itself on.
Your win rate is wins divided by consistent comparisons — not total comparisons. A high consistency rate means the judge is stable and the win rate is trustworthy. A low one is itself a finding: the judge is too position-biased to use on this task.
None of the above means anything if the judge disagrees with people. Before you trust a single debiased win rate, run the judge on a sample your team labeled by hand and confirm high agreement — exactly the rater-vs-gold check from Week 3, pointed at a model.
The fraction of comparisons that survive the swap is a number you report. If 90% are consistent, your judge is solid and your win rate is defensible. If only 50% survive, half your eval was decided by slot order — you don't quietly drop them and report the rest as if nothing happened; you flag that the judge is unfit for this task and escalate to human eval. The discards are a quality signal about the judge, not noise to hide.
Open the "Try This" box below and type in llm_judge.py. Run it, confirm it reports consistent comparisons and a debiased win rate, then do the work that makes it real:
judge() with a real model call if you can (honors). The control logic doesn't change — that's the point of building it first.Keep the output. The win rate plus the judge-vs-human agreement number is a portfolio-grade artifact: it proves you can run an eval and prove the eval is honest.
A tutor runs the judge and reports the win rate. A lead reports the win rate plus the consistency rate plus the judge-human agreement — three numbers that together say "this result is trustworthy, and here's the evidence." Never hand up a win rate naked. The controls are the credibility.
A tutor reports a win rate. A lead reports the win rate plus the consistency rate plus the judge-human agreement — and treats the discarded, order-flipped verdicts as a finding about the judge, not noise to quietly drop. The controls are the credibility; a naked win rate is an assertion, not a measurement.
Say this in an interview: "I never hand up a win rate alone. I run every comparison twice with the order swapped, score only the self-consistent verdicts, and report the consistency rate — if half the verdicts flip on slot order, that's not a result, it's a signal the judge is unfit for the task."