Telling a real quality drop from random noise — with a confidence interval, an alert threshold, and a trend line
Day 42 of 66
Yesterday you built the loop. Today we make its measurement honest. Here's the trap that catches new leads: your weekly audit comes back at 88% one week, 84% the next. Did quality drop four points — or did you just happen to sample a slightly harder set of items? If you raise an alarm every time the number wiggles, you'll cry wolf and exhaust your team. If you ignore the wiggles, you'll miss the real decline. The skill is telling the two apart.
Any accuracy you compute from a sample is an estimate with uncertainty baked in. A single number like "88%" is meaningless without its error bar. The lead's job is to know how wide that error bar is, and to only sound the alarm when a change is bigger than the noise.
You met sampling error in Week 3. Now you operationalize it. When you check 100 sampled labels against gold and 88 are correct, your measured accuracy is p = 0.88. But the true accuracy of the whole label set could plausibly be a bit higher or lower — the sample is just a peek. A rough 95% confidence interval tells you the plausible range.
For an accuracy rate p measured on n sampled items, the rough 95% confidence interval is:
p ± 1.96 · √( p(1−p) / n )
The term under the root is the standard error — how much your sample estimate jitters around the truth. The 1.96 turns one standard error into the ~95% band (the standard normal cutoff). The whole point: bigger n shrinks the interval. A sample of 100 gives a band of roughly ±6–7 points; to halve that band you'd need 4× the sample (≈400), because n is under a square root.
So "88% on 100 items" really means "true accuracy is plausibly somewhere around 82%–94%." That width is exactly what stops you from over-reacting to a four-point wiggle.
Now compare two weeks. Their confidence intervals will overlap or they won't. That overlap is your signal.
| Pattern | What it means | Action |
|---|---|---|
| Intervals overlap heavily | The change is within noise — could be luck | Watch; don't alarm |
| Intervals barely touch / separate | A real change beyond sampling noise | Investigate the cause |
| Three weeks declining in a row | A trend, even if each step is small | Investigate — direction beats size |
One week dipping inside the noise band is nothing. Three weeks marching the same direction — 92%, 88%, 84% — is drift, even though no single step is dramatic. Always plot the trend, never judge a week in isolation. The line tells you what any single point can't.
You need a rule that fires automatically so you're not eyeballing every week. The rule has to be wide enough to clear the noise but tight enough to catch real decline. A defensible default:
Alert when this week's accuracy drops more than ~5 points below your baseline. Why 5? Because with a weekly sample of ~100, the noise band is roughly ±6–7 points, so a drop bigger than ~5 points is starting to exceed what luck alone explains — especially if it persists. Tie the threshold to your actual sample size: smaller samples need a wider threshold, larger samples a tighter one. Never pick a number without a reason; "5 points" without "because n=100" is just superstition.
This is the exact math the audit tool (tomorrow's build) runs on each week. Read it once here so the build day is about using it, not deriving it.
import math
def accuracy_ci(correct, sampled):
p = correct / sampled # measured accuracy this week
se = math.sqrt(p * (1 - p) / sampled) # standard error of that rate
lo = p - 1.96 * se # 95% interval, low end
hi = p + 1.96 * se # 95% interval, high end
return p, (round(lo, 3), round(hi, 3))
p, ci = accuracy_ci(88, 100)
print(f"accuracy {p:.0%} 95% CI {ci}") # accuracy 88% 95% CI (0.817, 0.943)Three lines do the work: divide for the rate, square-root for the standard error, then ±1.96·SE for the band. Everything else this week — drift alerts, the weekly report — is built on this tiny function. When you read "88% (±6 pts)" in a report next week, that's where the ±6 came from.
An evaluator reports "84% this week" and reacts to every wiggle. A lead reports the number with its error bar and only sounds the alarm when a change clears the noise — turning a jittery metric into a decision rule the whole team can trust.
Say this in an interview: "A sampled accuracy is meaningless without its confidence interval, so I tie my drift threshold to sample size — at n≈100 the band is ±6–7 points, which is exactly why a sub-5-point dip is noise and a three-week slide is signal."
p ± 1.96·√(p(1−p)/n).