Turn yesterday's three dials into running code: a queue builder that stratifies, injects hidden gold, and assigns consensus by difficulty
Day 38 of 66
You can describe sampling, consensus, and gold in prose all day. But the moment you write the builder, the fuzzy parts get sharp: exactly how many gold items at 10%? Exactly which items get N = 3? Exactly how do you shuffle so gold can't be spotted? Code forces every decision to be concrete — and a concrete pipeline is one you can defend, hand off, and instrument later.
The full pipeline.py is in the Try This box below (injected for build days). Type it in rather than pasting — reading every line is how the design lands. The code is small on purpose: the value is in what each line decides, not in its length.
Your build_queue function takes the real items plus a pool of gold items, and returns one shuffled work list. Four jobs, in order:
Pull items so each task type is represented. In the starter code the items already carry a task field ("safety" vs "quality"); a richer version would sample a target count per stratum. The point: the queue is composed on purpose, not by accident.
Each item gets a consensus count. Hard or risky tasks (here, anything in hard_tasks like "safety") get 3; everything else gets 1. This is yesterday's "spend consensus where error is expensive," made literal.
Compute n_gold = int(len(items) * gold_rate) and pull that many known-answer items from the gold pool. Each gold item is tagged is_gold: True and carries its known answer so you can score against it later. The real items are tagged is_gold: False.
One line — random.shuffle(queue) — but it's the line that makes gold work. After the shuffle, a gold item is positionally indistinguishable from a real one. Skip this and your quality measurement is a lie (see yesterday's warning).
A builder that returns a list isn't done — a lead always reports what they built. Before any annotator touches it, you should be able to state three numbers about the queue:
sum(x['is_gold'] for x in queue). ("Will I have enough gold hits to measure accuracy?")sum(x['consensus'] == 3 for x in queue). ("How much of my budget went to high-confidence items?")Those three numbers are your three dials, read back. If the gold count is 2, your 10% rate isn't buying enough signal — raise it or accept the uncertainty. If 80% of items need 3-way consensus, your cost just tripled — is that intentional or a mis-set difficulty rule? Printing the composition turns an opaque list into a design you can inspect and defend.
Run the script, then change one dial at a time and watch the report move. This is the whole point of having it in code.
gold_rate from 0.1 to 0.25. The gold count jumps — you bought more measurement, but those slots aren't producing new labels. Feel the cost."quality" to hard_tasks. Now nearly everything is N = 3 and your effective cost roughly triples. That's the throughput price of consensus from yesterday, live.Swap the toy items for a task you actually run as a tutor. Set hard_tasks to whatever genuinely deserves 3 eyes in your domain (text-in-image? faces? anything safety-adjacent?). Set gold_rate to a number you can defend. The output is now a real queue spec for a real task — the seed of this week's portfolio artifact.
A tutor describes a good pipeline in prose. A lead writes the builder — because code forces every fuzzy decision to be concrete (exactly 10% gold, exactly which items get N=3) and then reports the composition back so the design is auditable, not opaque.
Say this in an interview: "When I build a queue I print its composition — size, gold count, how many items need 3-way consensus — because those three numbers are my three dials read back. If they're wrong, I can see it before a single annotator touches the work."
pipeline.py does four jobs: stratify items → assign consensus by difficulty → inject gold at a target rate → shuffle to hide it.