Motion graphic

Generative AI Advanced Fine-Tuning for LLMs

9 hours · 2 modules · Hugging Face TRL (Transformer Reinforcement Learning)

The most mathematically demanding course in the certificate, and the most directly relevant to human-data work: this is alignment — how a raw next-token predictor becomes an assistant that follows instructions and behaves the way people prefer.

---

The three-stage alignment pipeline

Everything in this course sits in one pipeline. Fix it in your head before the details:

1. PRETRAINING          raw text, next-token prediction        → a model that completes text
2. INSTRUCTION TUNING   (instruction, response) pairs, SFT     → a model that follows instructions
3. PREFERENCE TUNING    human preference pairs, RLHF or DPO    → a model that responds the way people prefer

Stage 2 is supervised. Stage 3 is where the interesting problems live: it uses comparisons rather than gold answers, because for open-ended generation there is no single correct output — but humans can reliably say which of two responses is better.

That distinction — rating is hard, ranking is easy — is the foundation of the entire field, and of the Human Data Lead role.

---

Module 1 — Different Approaches to Fine-Tuning

Instruction tuning (SFT)

Also called Supervised Fine-Tuning. Train on (instruction → response) pairs so the model learns the shape of following an instruction, not just continuing text.

Before: "Explain gravity." → the model might continue with more exam questions, because that's what such text is usually followed by on the internet. After: it explains gravity.

Instruction tuning doesn't add knowledge. It changes behaviour — it teaches the model that an instruction should be answered.

Data format — the templating is what does the work:

### Instruction:
Summarise the following article.

### Input:
<article text>

### Response:
<the summary>

Or chat-formatted:

messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "Explain gravity simply."},
    {"role": "assistant", "content": "Gravity is..."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False)

Every model has its own template — Llama, Mistral and Qwen use different special tokens. Use apply_chat_template() rather than hand-formatting; the wrong template degrades quality with no error.

Loss masking: compute loss only on the response tokens, not the prompt. You want the model to learn to produce responses, not to reproduce instructions.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=SFTConfig(output_dir='./sft', num_train_epochs=3,
                   per_device_train_batch_size=4, learning_rate=2e-5,
                   max_seq_length=512),
    peft_config=lora_config,          # PEFT and SFT compose
)
trainer.train()

Best practices from the reading:

• Quality over quantity. ~1,000 excellent examples beat 100,000 mediocre ones — the LIMA result.
• Diversity of task type matters more than volume within a task.
• Consistent formatting; inconsistency teaches the model that format is arbitrary.
• Include refusals and edge cases, or the model learns it must always answer.
• 1–3 epochs. More overfits and degrades general capability.
• Hold out a set and evaluate qualitatively, not only on loss.

Well-known instruction datasets: FLAN, Alpaca (self-instruct generated), Dolly (human-written), OpenAssistant.

Reward modelling

The problem: for "write a poem about the sea", there's no single correct answer to supervise against. So instead of gold answers, collect comparisons — annotators see two responses and pick the better one — and train a model to predict those preferences.

Architecture: take a pretrained LM, replace the token-prediction head with a scalar head. Input = (prompt, response), output = one number: quality.

from trl import RewardTrainer, RewardConfig
from transformers import AutoModelForSequenceClassification

rm = AutoModelForSequenceClassification.from_pretrained('base-model', num_labels=1)

num_labels=1 — a regression head producing a single score.

The loss — Bradley-Terry, the model of pairwise comparison:

P(chosen ≻ rejected) = σ( r(x, y_chosen) − r(x, y_rejected) )

Loss = −log σ( r(x, y_c) − r(x, y_r) )

Read it plainly: only the difference in scores matters. The loss pushes the chosen response's score above the rejected one's. Absolute values are meaningless and unbounded — a reward model's raw output has no interpretable scale, which is why you normalise it in practice.

Dataset format: {prompt, chosen, rejected}. That's the standard HF preference-dataset schema and you'll see it again in DPO.

Why comparisons rather than ratings: inter-annotator agreement on "rate this 1–10" is poor — people anchor differently, drift over a session, and use the scale inconsistently. Agreement on "which of these two is better" is much higher. This is the single most important practical fact in preference data collection.

Reward hacking / Goodhart's law: the reward model is a proxy for human preference, not human preference. Optimise it hard enough and the policy finds inputs that score highly and are actually bad — excessive length, sycophancy, confident-sounding hedging. Every RLHF system needs a brake on this, which is what the KL penalty below is.

---

Module 2 — RLHF, PPO and DPO

LLMs as distributions, and as policies

The reframing the module opens with, and the one that makes the RL machinery make sense:

An LLM defines a probability distribution over next tokens: π(token | context).

In RL terms:

• State = the context so far (prompt + generated tokens)
• Action = the next token
• Policy π = the language model
• Reward = the reward model's score on the finished response

So generating a response is rolling out a policy in an environment, and improving the model is policy optimisation. Nothing changes about the model — only the framing.

RLHF

1. SFT model            (starting policy, from Module 1)
2. Reward model         (trained on human comparisons)
3. RL optimisation      (improve the policy against the reward model, with a leash)

The objective:

maximise   E[ r(x, y) ]  −  β · KL( π_current ‖ π_reference )
           └ reward ┘        └──── the leash ────┘

The KL penalty is the critical term. It measures how far the policy has drifted from the original SFT model and penalises drift. Without it:

• The policy collapses onto whatever exploits the reward model
• Output diversity dies
• Fluency degrades — the model discovers weird high-reward text that isn't language

β controls the trade-off. Too high → nothing changes. Too low → reward hacking and gibberish.

PPO — Proximal Policy Optimization

The RL algorithm used for the optimisation step. "Proximal" = stay close to the previous policy.

Why not plain policy gradient: the vanilla estimator has enormous variance, and a single large update can destroy the policy irrecoverably. There's no "reload the last good checkpoint" inside a training run that's already diverged.

PPO's clipped objective:

ratio  = π_new(a|s) / π_old(a|s)

L = min( ratio · Â ,  clip(ratio, 1−ε, 1+ε) · Â )

where  is the advantage — how much better this action was than the baseline expectation. ε is typically 0.2.

Plain English: if the new policy wants to change the probability of an action by more than ±20% in one update, clip the benefit. The model can still improve, but only in small steps. It's a trust region enforced by clipping instead of by constrained optimisation.

The four models in memory during PPO — this is why RLHF is expensive, and it's a favourite exam item:

1. Policy — being trained
2. Reference — frozen SFT copy, for the KL term
3. Reward model — scores responses
4. Value model / critic — estimates expected return, for computing advantages

Four models of comparable size. That's the practical objection to PPO.

from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead

config = PPOConfig(model_name='gpt2', learning_rate=1.41e-5, batch_size=64,
                   init_kl_coef=0.2)
policy = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
ref    = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')

trainer = PPOTrainer(config, policy, ref, tokenizer)

for batch in dataloader:
    query_tensors    = batch['input_ids']
    response_tensors = trainer.generate(query_tensors, **gen_kwargs)
    rewards          = reward_model(query_tensors, response_tensors)
    stats            = trainer.step(query_tensors, response_tensors, rewards)

AutoModelForCausalLMWithValueHead — the LM plus a scalar value head, so policy and critic share a trunk.

The log-derivative trick (in the readings) is the identity that makes policy gradients computable:

∇θ E[f(x)] = E[ f(x) · ∇θ log πθ(x) ]

Because ∇log π = ∇π/π. It converts a gradient of an expectation over a distribution you're differentiating into an expectation you can estimate by sampling. It's the mathematical foundation of REINFORCE and everything built on it.

DPO — Direct Preference Optimization

The result the module builds to: you can skip the reward model and the RL loop entirely.

The derivation, in three moves:

1. The constrained objective has a closed-form optimum. Maximising reward minus β·KL is

solved analytically by: `` π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r(x,y) / β ) `` Z(x) is the partition function — the normalising constant summing over all possible responses. Intractable to compute, which is why people used RL.

1. Invert it. Solve that expression for the reward:

`` r(x,y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x) `` The reward is expressible in terms of the policy itself. There is no separate reward function needed — the policy implicitly defines one.

1. The partition function cancels. Substitute into the Bradley-Terry loss. Because that loss

depends only on the difference between two rewards for the same prompt x, and β·log Z(x) is identical for both, it cancels exactly.

The DPO loss:

L = −log σ( β·log[πθ(y_c|x)/π_ref(y_c|x)] − β·log[πθ(y_r|x)/π_ref(y_r|x)] )

In words: increase the policy's probability of the chosen response relative to the reference, and decrease it for the rejected one. It's a plain classification loss on preference pairs. No sampling, no reward model, no critic, no RL.

Why this matters — the practical comparison:

That last row is the real remaining argument for PPO: online RL can discover responses better than anything in the preference dataset, because it generates and scores new samples. DPO can only reweight what it's shown.

from trl import DPOTrainer, DPOConfig

trainer = DPOTrainer(
    model=model,                  # the SFT model, being trained
    ref_model=ref_model,          # frozen copy (omit when using PEFT — the base acts as ref)
    args=DPOConfig(output_dir='./dpo', beta=0.1, learning_rate=5e-7,
                   per_device_train_batch_size=4),
    train_dataset=dataset,        # {prompt, chosen, rejected}
    processing_class=tokenizer,
)
trainer.train()

Note the learning rate: 5e-7 to 1e-6, far lower even than SFT. DPO is sensitive and over-training it degrades the model — a documented failure mode where likelihood of both chosen and rejected responses falls.

β ≈ 0.1 typically. Higher β stays closer to the reference; lower β allows more drift.

With PEFT, you don't need a separate reference model — disabling the adapters recovers the base model, so TRL uses that. Halves memory again.

Successors worth knowing by name: IPO (fixes a DPO overfitting failure mode), KTO (needs only thumbs-up/down, not pairs), ORPO (merges SFT and preference tuning into one stage), GRPO (group-relative, used in recent reasoning models).

Constitutional AI / RLAIF

Mentioned in passing: replace human comparison labels with AI-generated ones, using a written set of principles ("a constitution") to guide the judging model. Scales preference data collection enormously; the trade-off is that the judge's biases become the system's biases.

Relevant to your xAI track: this is exactly the boundary where the human-data question becomes "which comparisons genuinely need a human, and which can be modelled?"

InstructLab

IBM's open-source alignment tooling (with Red Hat). The pitch: contribute knowledge and skills as structured YAML taxonomies, use a teacher model to synthesise training data from a small number of human examples, then fine-tune locally. Positioned as community-driven, incremental model improvement without a full retrain.

Course 13's reading compares InstructLab fine-tuning against RAG — same trade-off as before: fine-tune for skill and behaviour, RAG for facts.

---

Course 11 in one page

• Pretrain → instruction-tune (SFT) → preference-tune (RLHF or DPO)
• SFT changes behaviour, not knowledge; mask the loss to the response; use the model's own chat template
• Quality beats quantity in instruction data (~1k great examples)
• Reward model = LM + scalar head, trained with Bradley-Terry on {prompt, chosen, rejected}
• Only score differences matter; comparisons beat ratings because agreement is higher
• Reward hacking is Goodhart's law: the proxy is not the goal
• RLHF objective = reward − β·KL(π ‖ π_ref); the KL term is the leash
• PPO clips the probability ratio to ±ε (0.2) so no single update wrecks the policy
• PPO needs 4 models: policy, reference, reward, value
• DPO derives the reward from the policy itself; the partition function cancels in the pairwise

loss; 2 models, no RL, no reward model, offline

• DPO learning rate is tiny (5e-7); β≈0.1; over-training degrades both chosen and rejected likelihoods
• PPO's remaining edge: it can surpass the preference dataset because it explores