Build the tool that tells you, every week, whether your data is getting better or worse
Day 43 of 66
Day 41 gave you the loop. Day 42 gave you the math. Today you turn both into a tool you'd actually run every Monday: take a sample, compare it to gold, get an accuracy with an honest error bar, and have the script tell you itself whether quality drifted. The "Try This" box below contains the full qa_audit.py — type it in, don't paste, so you read every line.
qa_audit.py is a real piece of your portfolio and a real tool you could run on the job day one. It's small on purpose — a weekly audit that takes 200 lines never gets run. The discipline is in the cadence and the interpretation, not the code length.
Before you build it, hold the four moves in your head. Each maps to something you already learned this week.
Each week you pass in two numbers: how many sampled items were correct against gold, out of how many you sampled. That's the raw measurement — accuracy on a fresh random slice (Day 41's cadence).
The script computes p = correct/sampled and the 95% interval p ± 1.96·√(p(1−p)/n) (Day 42's formula). The interval is what stops you treating a noisy wiggle as a crisis.
It compares the latest week to the baseline week. If accuracy has dropped more than your threshold (≈5 points), it prints a DRIFT ALERT — the automatic rule from Day 42 so you're not eyeballing.
It prints every week in sequence so you see the line, not a single point. Three weeks sliding the same direction is the signal, even when each step is inside the noise.
The "Try This" box below has the complete qa_audit.py. It defines an audit() that returns the week's accuracy and CI, runs it over a few weeks of history, prints the trend, and fires a drift alert when the latest week falls more than 5 points below the first. Type it, run it, then change the numbers and watch the alert turn on and off.
Feed it three flat weeks (90/100 each): no alert, as it should be. Now feed it 92, 88, 81: the alert should fire. Then feed it 92, 88, 90 — a wiggle, not a trend — and confirm it stays quiet. You're testing that your threshold separates drift from noise, exactly the judgment from Day 42.
An alert tells you that quality fell, never why. The "why" is the lead's actual work, and it almost always traces to one of two recent changes. Check them in this order.
| First suspect | The question to ask | If yes, the fix |
|---|---|---|
| Recent guideline changes | Did the rubric change right before the drop? A "clarification" often introduces a new ambiguity. | Revert or sharpen the change; add worked examples for the cases it confused. |
| New annotators | Did anyone start recently? New people are uncalibrated until proven otherwise. | Pull their numbers out separately; route them through calibration (Week 10) before live work. |
| Input distribution shift | Did the kind of items being labeled change (new model, new prompt type)? | The guideline may not cover the new cases — extend it; adjudicate the new patterns. |
| A single drifting veteran | Is one experienced annotator quietly shortcutting under load? | Per-person agreement check; a calibration conversation, not a blanket retrain. |
When accuracy drops, your first two questions are always "what changed in the guideline?" and "who's new?" — because those are the cheapest things to check and the most common culprits. Slicing the audit by annotator and by week of a guideline edit turns one scary aggregate number into a specific, fixable cause. That slicing is the whole game.
qa_audit.py from the Try This box and run it on the provided weeks. Confirm the accuracy, CI, and drift alert all print sensibly.An annotator sees a scary aggregate — "accuracy fell to 83%" — and braces for a retrain. A lead slices that number by annotator and by guideline-edit week until it names a specific, cheap cause: the rubric clarification that backfired, or the one new hire who isn't calibrated yet.
Say this in an interview: "When an audit fires a drift alert, my first two questions are always 'what changed in the guideline?' and 'who's new?' — because slicing the data by those two turns one frightening aggregate into a fixable cause faster than touching the model ever would."
qa_audit.py = sample vs gold → accuracy + rough CI → week-over-week drift alert → readable trend.