Follow one prompt from base model to aligned assistant — and see exactly where a human is required at each step
Day 2 of 66
Day 1 gave you the three sources of behavior. Today we run the actual assembly line that turns them into a shippable assistant. This is the post-training pipeline, and InstructGPT's three steps are still the canonical recipe every lab riffs on.
Humans write ideal answers to a set of prompts. The base model is fine-tuned to imitate them. Output: a model that behaves like an assistant, but whose quality is capped by how good and how consistent the human demonstrations were.
For a prompt, the model generates several responses. A human ranks them best-to-worst. These rankings train a separate reward model — a model whose only job is to predict "how much would a human like this output?" The RM is a learned, automatic stand-in for human taste.
The assistant (the "policy") is then improved with reinforcement learning to produce outputs the reward model scores highly — while a constraint keeps it from drifting too far from the SFT model. The result is a model tuned toward human preference at scale, without a human in the loop for every output.
Humans can't rate billions of outputs during training. So we use human comparisons to train a reward model once, then let that reward model supervise the billions. Your comparisons don't grade the final model — they teach the grader. That's why comparison quality compounds: every error in your preference data becomes a systematic bias in the grader, which becomes a bias in the model.
Take the prompt "a cinematic wide shot of a city at dawn." Here's the human touch at each stage:
| Stage | What happens | Human data required |
|---|---|---|
| Pretraining | Model learns what "cinematic," "dawn," and "wide shot" mean from captioned images/text | Curation & filtering of the scraped corpus |
| SFT | A tutor writes/marks an exemplar good generation + caption for this kind of prompt | A demonstration (your gold example) |
| Reward model | Model makes 4 images; a tutor ranks them on adherence + aesthetics | A ranked comparison (your preference labels) |
| RLHF | Model is nudged to generate images the RM scores high | None directly — but quality is inherited from your labels |
| Evaluation | Before shipping, humans judge new-vs-old on a held-out prompt set | Evaluation ratings (Week 6) |
The three-step recipe is the foundation, but the field moved fast. You don't need the math — you need to recognize the names, because they change what kind of data you collect:
You will be tempted to dive into PPO, KL penalties, and RL theory. Resist this week. You need the shape of the pipeline and the role of human data in it, not the optimizer internals. The "Not This Week" box below lists exactly what to skip.
An evaluator collects clean comparisons. A lead decides whether the goal even needs preference data — versus SFT or eval — because each is a different product with a different cost and quality bar. Knowing that DPO makes pairwise data central, and that Constitutional AI shifts the work toward writing principles, is how you spec a collection that doesn't waste the budget.
Say this in an interview: "Before I spec a collection I ask which data product the goal needs — SFT, preference, or eval — because each has a different cost structure and quality bar, and picking the wrong one burns the budget before a single label is wrong."