Add the dataset plan, the weekly QA and drift alerts, and the calibration plan — then define the eval that will prove the program actually worked
Day 58 of 66
You have a scoped problem and a consistent core. Today you build the operating layer: the dataset plan, the weekly quality loop, the calibration plan, and — most important — the eval that will prove the program worked. The Try This box below contains capstone_brief.py; running it prints the spine of your program as a single dictionary where every line points at an artifact you built. Type it in, fill it with your problem's values, and run it. That printout is the skeleton this whole document hangs on.
By now the code is the easy part — you've written six of these scripts. The point of capstone_brief.py isn't the Python; it's that seeing your entire program collapse into seven labeled lines forces you to notice anything missing. If a line says "TODO," that's the part of your program that isn't real yet.
Spell out the data you'll collect, in the format a training team consumes, targeted at the failure the program is fixing.
A director will ask "and this gets us to 85% how?" Answer with a chain, not a hope: "The Week 7 analysis shows 60% of adherence failures cluster in multi-object prompts; 1,500 targeted preference pairs on that cluster, at our historical conversion of label-quality to model-quality, projects a ~10–13 point lift — enough to clear 85% with margin." You won't be exactly right. You'll be credible, because you reasoned from the data instead of guessing.
Bring in your Week 9 audit as the program's heartbeat. Every week, sample labels, compare to gold, compute accuracy with a confidence interval, and fire a drift alert if accuracy drops more than a set threshold.
Attach your Week 10 playbook so the people stay aligned as the program runs. State the cadence, what triggers a session, and how the agreement report drives the agenda.
This is the part that turns a program into a result. Define, in advance, the evaluation that will answer "did it work?" — before you've seen any outcome, so the answer can't be rationalized after the fact.
Decide now exactly what reading counts as success and what counts as failure. "We hit 85% gold-set accuracy and a >60% human-eval win-rate against the current model on a held-out set" is a pre-registered verdict. Defining it before the data arrives is what makes an eval honest — it removes your ability to move the goalposts when the number is disappointing.
The cheap automated reading (gold-set accuracy) runs weekly and shows the trend. The expensive trusted reading (human eval, with the Week 6 position-bias controls) runs at the milestone and is the verdict. If the cheap and expensive readings disagree, the expensive one wins — and the gap is itself a finding to investigate.
Everything upstream — rubric, pipeline, datasets, QA — exists so this one eval can give an honest answer. When you can say "here is the held-out eval, here is the win-rate I pre-registered, here is the number we hit," you've stopped describing work and started reporting a result. That sentence is what a director remembers.
capstone_brief.py from the Try This box with your problem's real values. Any line reading "TODO" is a gap — close it.An annotator hopes the data helps. A lead pre-registers the verdict: states the exact gold-set accuracy and human-eval win-rate that count as success before any result exists, and projects the expected lift by reasoning from the failure-cluster data instead of guessing — credible, not lucky.
Say this in an interview: "I define what counts as success before the data lands, so I can't move the goalposts when the number disappoints. And when a director asks 'how does this get us to 85%,' I answer with a chain from the cluster analysis, not a hope."