A lead has touched the tools — you'll stand up a real labeling project before you ever claim you can design one
Week 8 taught you the pipeline as a set of decisions — sampling, consensus, gold injection, review tiers. That thinking is the job. But there's a sentence that quietly sinks lead candidates in interviews: "I understand pipelines conceptually." It loses, every time, to "I stood up a labeling project last week — here's the config, here's where the UI fought my annotators."
The reason is simple. A lead designs how work flows, but a lead also has to spec the tool, debug why agreement is low, and tell an engineer exactly what the platform can and can't do. None of that survives contact with a candidate who has only read the docs. You don't need to be a power user of all three platforms. You need muscle memory in at least one — a project you built, a config you wrote, an export you compared to a gold key — so that "I could set this up" is a memory, not a hope.
By the end you will have a running Label Studio project on your own machine, scoring six real image pairs on prompt-adherence, exported to a file you compare against the gold answers. That artifact — not the reading — is the point. A lead has reps.
You'll meet four serious annotation platforms plus a video/multimodal specialist. The trap is treating this as a tool tutorial. It isn't. A lead picks a tool by task type and build-vs-buy — then maps the same pipeline onto whatever the team already runs. Here's the map you actually need.
| Platform | Best for (task type) | Open-source vs commercial | Multimodal / video strength | Build vs buy |
|---|---|---|---|---|
| Label Studio | General-purpose: text, image, audio, video, time-series, and LLM/RLHF rating & preference tasks | Open-source (free Community edition); self-hostable | Broad — handles image and video labeling, but not video tracking specialist | Build — your default open-source choice |
| CVAT | Computer vision: detection, segmentation, keypoints, and object tracking across video frames | Open-source (originally Intel, now stewarded by CVAT.ai); self-host via Docker or use hosted | Strongest open-source video — frame-by-frame annotation with interpolation | Build — when the task is CV / video specifically |
| Labelbox | Managed labeling at scale with built-in QA workflows, enterprise MLOps hooks, active learning; also runs an expert-rater network (Alignerr) | Commercial SaaS (~$1.5k/mo to $50k+/yr tiers) | Good multimodal coverage, model-assisted labeling, supports external workforces | Buy — when you want managed quality infrastructure, not self-hosting |
| Encord | Video-native and multimodal annotation (incl. medical/DICOM) with consensus scoring built into the product | Commercial SaaS | Strongest commercial video — long-form video tooling + automated consensus/QA | Buy — when video is the core task and you want QA out of the box |
| Argilla | LLM feedback: preference, rating, and SFT-shaped text data | Open-source; purpose-built for LLM data | Text-first (not a CV/video tool) | Build — when the data is preference/rating, as in Weeks 4 & 6 |
Ask two questions in order. (1) What's the task? If it's video tracking or segmentation → CVAT (build) or Encord (buy). If it's LLM preference/rating → Argilla. If it's mixed or general → Label Studio. (2) Build or buy? If your team wants to own the stack and iterate fast, self-host the open-source option. If they want managed consensus, benchmarks, and review queues without running servers, that's the case for Labelbox or Encord. The pipeline you designed in Week 8 maps onto all five — the tool just changes who runs the infrastructure. One 2026 wrinkle a lead should know: since Meta took 49% of Scale AI, vendor neutrality is a real procurement criterion — several labs walked from Scale's stack over it. If you're the one specing the tool, "who else owns this vendor" is now a legitimate question to raise.
Two shifts since this track was first written. Model-assisted labeling is now table stakes — every serious platform pre-labels with a model and puts humans in the reviewer/verifier seat, which moves the lead's job from "design the labeling" toward "design the verification": what does the human check, in what order, against what gold? Say it that way in interviews. And the platforms are converging on built-in QA — consensus scoring, benchmarks, and review queues shipped as product features (Encord and Labelbox especially), so "we'd have to build QA" is less often true. The task model below is unchanged — it's exactly why these shifts are easy to absorb.
Here is the leverage. Every one of these tools is a different UI over the same underlying model. Learn the model and any new platform is thirty minutes away, because you already know what you're looking for in its menus.
One unit of work: an item plus the question asked about it. "Rate this image's adherence 1–5." "Which of A or B is better?" In Label Studio a task is one row of imported data; the config defines the question.
The people (or models, in pre-labeling) who answer. The platform tracks who answered what — the precondition for measuring anything.
How many annotators see the same task. One is fast and cheap; multiple-with-a-vote catches individual error. Every platform exposes this as a per-project setting, sometimes called overlap or annotations-per-task.
Known-answer items mixed invisibly into the queue so you measure accuracy live. Label Studio injects them as tasks with a pre-set answer; Labelbox calls these benchmarks. Same idea, different label.
An escalation path for uncertain or high-stakes labels — a second set of eyes before the label is trusted. Platforms model this as a review queue or QA stage.
The measured consistency across annotators (and against gold). Low agreement is a signal: ambiguous rubric, confusing UI, or a genuinely hard item. The platform surfaces it; you diagnose it.
When an interviewer asks "we use a tool you've never touched — could you ramp?" the honest, strong answer is: "Yes. Show me where it sets consensus, where gold lives, and where the review queue is, and I'm productive in an afternoon. The model is the same; only the buttons move." That answer is only credible if you've actually built one project. So build one.
Most lead candidates come from text/NLP and treat video as "images, but more of them." You know better. Video has temporal structure — an object that's correct in frame 1 can drift, occlude, or flicker by frame 90, and that's exactly the failure surface a generation team cares about.
CVAT is built for this. It does object tracking across video frames with interpolation: you annotate a bounding box or keypoint on a keyframe, CVAT interpolates it across the intervening frames, and you correct where the model's guess drifts. That is the same eye you use grading a shot — watching a subject hold (or fail to hold) across a cut. When the task is "annotate motion consistency in generated video," you don't just understand the tool; you understand the artifact the tool produces. Lean on that. It's the part of platform fluency the NLP-native leads can't fake.
Separately from today's Label Studio build, spend thirty minutes in CVAT (hosted at cvat.ai or self-hosted via Docker): import a short clip, draw one bounding box on the first frame, let it track/interpolate, then scrub and fix the drift. One clip is enough to say "I've tracked objects across video frames in CVAT" — and mean it.
Now the real rep. Label Studio is the right default for today because it spans rating and pairwise tasks in one tool, runs locally in two commands, and models consensus and review the way Week 8 described. Install and start it.
$ pip install label-studio $ label-studio start
That opens the app in your browser (default http://localhost:8080). Create an account locally, then Create Project. The thing that makes a project a project is its labeling config — an XML-like spec of what the annotator sees and what they can input. You paste it under Labeling Interface → Code. Here are two realistic, minimal configs for the work you actually do.
One image, the target prompt shown as context, a 1–5 rating, and a free-text field for deviations. This mirrors the gold schema in the lab dataset.
<View>
<Header value="Score how well the image matches the target prompt"/>
<Text name="prompt" value="$target_prompt"/>
<Image name="img" value="$image"/>
<Rating name="adherence" toName="img"
maxRating="5" icon="star" size="medium"/>
<TextArea name="deviations" toName="img"
placeholder="List wrong/missing elements"
rows="3"/>
</View>Two images from the same brief, one choice. This produces the chosen/rejected signal you studied in Week 4 — directly comparable to the dataset's preference_pairs.jsonl.
<View>
<Header value="Which image better matches the target prompt?"/>
<Text name="prompt" value="$target_prompt"/>
<Pairwise name="pref" toName="imgA,imgB"
leftChoiceValue="A" rightChoiceValue="B"/>
<Image name="imgA" value="$image_a"/>
<Image name="imgB" value="$image_b"/>
</View>The $image, $target_prompt, $image_a, $image_b tokens are columns Label Studio expects in your imported data. If a task renders blank, the mismatch between your config's variable names and your import's keys is almost always why. This is the single most common first-build snag — and exactly the kind of thing a lead has hit before, not read about.
Use the practice dataset at ../assets/dataset/: six prompt-adherence image pairs in images/, target prompts in prompts.csv, and the gold answer key in labels.csv (adherence 1–5 + deviations). Do not open labels.csv until step 6.
label-studio start) and create a project. Paste Config A (image rating) into the labeling interface.../assets/dataset/images/. Add the matching target_prompt for each (pull it from prompts.csv — each pair shares one target). The fastest path is a small import file pairing each image with its target_prompt; or drag the images in and paste prompts as you go.labels.csv and compare. Count exact matches on the 1–5 score, and note any pair where your deviations missed something the gold caught. The B-variants are the deliberate low-adherence images; if you scored a B as a 4 or 5, that's a finding about your own rubric, not just the tool.A screenshot of your finished project (the annotation view with a scored item) plus the three-sentence tooling note. Drop both in your portfolio folder. That screenshot is the proof behind "I stood up a labeling project last week" — keep it where you can produce it in ten seconds during an interview.
Notice what you just exercised: a task (rate this image), annotators (you), a gold key (labels.csv), and an agreement check (your scores vs gold). That's four of the six task-model elements, hands-on, in one sitting. Add a second rater's column from sample_annotations.csv and you've touched consensus and agreement too — the full model, on real data.