Motion graphic

Generative AI Engineering and Fine-Tuning Transformers

8 hours · 2 modules · Hugging Face + PyTorch + PEFT

The most immediately employable course in the certificate. LoRA and QLoRA are what "AI Engineer" means on a job description in practice.

---

Module 1 — Transformers and Fine-Tuning

Hugging Face vs PyTorch

Not competitors — Hugging Face sits on top of PyTorch.

In practice: HF for the model and tokenizer, PyTorch when you need a custom loop. That's the combination the labs use.

The three tiers of the HF API:

# 1. pipeline — highest level
from transformers import pipeline
pipe = pipeline('sentiment-analysis')
pipe("This works.")

# 2. Auto classes — the normal working level
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok   = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

inputs  = tok("This works.", return_tensors='pt')
outputs = model(**inputs)
outputs.logits

# 3. Plain PyTorch — the model IS an nn.Module
for p in model.parameters(): ...

The AutoModelFor… suffix picks the task head: SequenceClassification, TokenClassification (NER), QuestionAnswering, CausalLM, MaskedLM, Seq2SeqLM.

Always load the tokenizer from the same checkpoint as the model. Mismatched vocabularies fail silently and produce plausible garbage.

Pretraining vs fine-tuning

The economics is the whole argument of the certificate: pretraining is unaffordable; adaptation is cheap; therefore the job is adaptation.

When to fine-tune vs prompt vs RAG

The decision the course sets up and Course 12 completes:

Simplest phrasing, and the one to use in interviews: fine-tuning teaches skills and style; RAG supplies knowledge. Do not fine-tune to inject facts — it's expensive, it fades, and it hallucinates around the edges.

Full fine-tuning with PyTorch

model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)   # note: 5e-5, not 1e-3

for epoch in range(3):
    model.train()
    for batch in train_loader:
        optimizer.zero_grad()
        out = model(input_ids=batch['input_ids'],
                    attention_mask=batch['attention_mask'],
                    labels=batch['labels'])
        out.loss.backward()                    # HF returns the loss when labels are passed
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

Two HF-specific conveniences: pass labels= and the model computes the loss for you; the output is an object with .loss and .logits, not a bare tensor.

Fine-tuning learning rates are 10–100× smaller than training-from-scratch rates. 2e-5 to 5e-5 is the standard band. Use 1e-3 and you will erase the pretrained knowledge — "catastrophic forgetting". This is the most common practical mistake and a certain exam question.

Typical recipe: 2–4 epochs, batch 16–32, AdamW, linear warmup over ~10% of steps then decay. More epochs overfits fast on small datasets.

Fine-tuning with the HF Trainer

from transformers import TrainingArguments, Trainer

args = TrainingArguments(
    output_dir='./results',
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    eval_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    fp16=True,
)

trainer = Trainer(model=model, args=args,
                  train_dataset=train_ds, eval_dataset=eval_ds,
                  tokenizer=tokenizer, compute_metrics=compute_metrics)
trainer.train()
trainer.evaluate()

Trainer handles the loop, mixed precision, checkpointing, logging, LR scheduling and multi-GPU. Use it unless you need something it can't express.

Freezing layers

Partial fine-tuning — the bridge to PEFT:

for param in model.bert.parameters():
    param.requires_grad = False              # freeze the encoder
for param in model.classifier.parameters():
    param.requires_grad = True               # train only the head

Cheap and fast, but limited: the frozen representations may simply not suit your domain. PEFT gets most of full fine-tuning's quality at similar cost — which is why it won.

---

Module 2 — Parameter Efficient Fine-Tuning (PEFT)

The problem

Full fine-tuning of a 7B-parameter model requires roughly:

• 14 GB for fp16 weights
• 14 GB for gradients
• 56 GB for Adam optimiser states (two fp32 moments per parameter)
• plus activations

≈ 84+ GB before activations — beyond a single consumer or even a single A100 40GB GPU. And you get a full 14 GB checkpoint per task.

PEFT trains <1% of the parameters and stores adapters of a few megabytes.

Three PEFT families

1. Additive — Adapters. Insert small bottleneck modules between existing layers; train only those.

h → down-project (d→r) → non-linearity → up-project (r→d) → + h

Works well. Downside: adds sequential layers, so inference gets slightly slower.

2. Reparameterisation — LoRA. Learn a low-rank update to existing weights. No added inference latency once merged. This is the one that matters.

3. Soft prompts — Prompt/Prefix/P-Tuning. Freeze the whole model and learn continuous "virtual token" embeddings prepended to the input. Extremely few parameters. Not human-readable — they're vectors in embedding space, not words. Weaker than LoRA on hard tasks but exceptionally cheap and easy to swap per task.

LoRA — Low-Rank Adaptation

The insight: the weight update learned during fine-tuning has low intrinsic rank. So don't learn the full ΔW — learn its factorisation.

W' = W₀ + ΔW = W₀ + B·A

W₀ : d × k   frozen
A  : r × k   trainable, initialised random Gaussian
B  : d × r   trainable, initialised ZERO
r  ≪ min(d, k)

Parameter count drops from d×k to r×(d+k). For d=k=4096, r=8: 16.7M → 65k. 256× fewer.

B is initialised to zero so that B·A = 0 at the start and the model begins exactly as the pretrained one — no random perturbation of a good model. If both were random, step 0 would already have damaged it. Guaranteed exam question.

Scaling: the update is applied as (α/r)·B·A. α is a constant that decouples the effective learning rate from the choice of r, so you can change r without retuning everything. Common convention: α = 2r, or α = 16 with r = 8.

Where to apply it: the attention projections. Original paper targets q_proj and v_proj; applying to all four (q,k,v,o) plus the MLP projections generally performs better at more cost.

At inference you can merge: W = W₀ + BA gives a single matrix and zero added latency — the decisive advantage over adapters.

from peft import LoraConfig, get_peft_model, TaskType

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622

Then train it exactly like any other model — Trainer or a plain loop.

model.save_pretrained('./lora-adapter')       # a few MB, not GB

from peft import PeftModel
base   = AutoModelForCausalLM.from_pretrained('base-model')
loaded = PeftModel.from_pretrained(base, './lora-adapter')
merged = loaded.merge_and_unload()            # fold into base weights for deployment

Choosing r: 4–8 for simple/stylistic tasks, 16–32 for harder ones, 64+ rarely helps and starts to overfit. Start at 8.

The practical superpower: one base model in memory, many task adapters swapped at runtime.

Quantization

Store weights in fewer bits.

Quantization maps a continuous float range onto a discrete grid. The error is the rounding residual. Doing it per-block rather than per-tensor keeps outliers from destroying the scale for everything else.

QLoRA

Quantized LoRA — the technique that put 65B fine-tuning on a single 48GB GPU, and 7B on a free Colab.

Recipe: freeze the base model quantized to 4-bit, and train LoRA adapters in 16-bit on top. Gradients flow through the frozen quantized weights to the adapters.

Three specific contributions, all quizzable:

1. NF4 (4-bit NormalFloat) — a quantization grid that is information-theoretically optimal

for normally-distributed weights, which is what neural network weights approximately are

1. Double quantization — quantize the quantization constants too; saves ~0.4 bits/parameter
2. Paged optimizers — page optimiser state to CPU RAM on memory spikes, preventing OOM

crashes on long sequences

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16,      # compute in bf16, store in 4-bit
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained('model-name', quantization_config=bnb,
                                             device_map='auto')

from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

Note bnb_4bit_compute_dtype — weights are stored in 4-bit and dequantized to bf16 for the actual matmul. Storage precision and compute precision are different things.

Trade-off: QLoRA is slower per step than LoRA (dequantization overhead) but uses far less memory. Reported quality is close to 16-bit LoRA.

Comparison

Ethical considerations in fine-tuning

The course devotes a video to this, and it maps directly onto the Human Data Lead work:

• Bias amplification — fine-tuning on a narrow dataset can concentrate biases that were

diluted in the pretraining mix

• Data provenance and consent — do you have the right to train on this? Copyright, licence,

and PII

• Safety regression — fine-tuning can strip a model's safety alignment, even unintentionally,

even on benign data

• Dual use — a model fine-tuned for persuasion is a model fine-tuned for manipulation
• Transparency — document the data, the intended use, and the known limitations (model cards)
• Evaluation beyond accuracy — test for fairness across subgroups, not just aggregate score

---

Course 10 in one page

• Hugging Face on top of PyTorch; AutoModelFor<Task> picks the head; tokenizer must match the checkpoint
• Fine-tuning LR = 2e-5 to 5e-5. Too high → catastrophic forgetting
• Pass labels= and HF computes the loss; output has .loss and .logits
• Fine-tuning teaches skill and style; RAG supplies knowledge
• PEFT families: additive (adapters) / reparameterisation (LoRA) / soft prompts
• LoRA: W' = W₀ + (α/r)·B·A; B starts at zero so training starts from the pretrained model
• r = 8 to start; target q_proj/v_proj; adapters are megabytes; merge for zero-latency inference
• Quantization: FP32→FP16→INT8→NF4; store low, compute in bf16
• QLoRA = 4-bit frozen base + 16-bit LoRA adapters; NF4 + double quantization + paged optimizers
• Fine-tuning can silently remove safety alignment — evaluate for it