Two lists. The first is what quizzes and final exams reliably ask. The second is code that runs fine and is wrong — the ones that cost you an evening.

---

Scaling: which algorithms need it

Need it: SVM, KNN, logistic regression, neural networks, PCA, K-Means, anything using distances or regularisation. Don't: decision trees, random forests, XGBoost — they split on thresholds, and monotonic rescaling doesn't change the ordering.

C is inverted

In sklearn's LogisticRegression and SVC, C = 1/α. Small C = strong regularisation. C=0.01 is heavily regularised; C=100 is barely regularised. In Ridge/Lasso it's alpha, the normal way round.

Bagging vs boosting

Bagging → reduces variance, parallel (random forest). Boosting → reduces bias, sequential (XGBoost).

L1 vs L2

L1/Lasso → drives coefficients to exactly zero (feature selection). L2/Ridge → shrinks toward zero, never reaching it.

Accuracy on imbalanced data

99% legitimate transactions → "always legitimate" scores 99% accuracy and catches nothing. Use precision, recall, F1, AUC, or a precision-recall curve.

Precision vs recall, decided by consequence

Missed detection is catastrophic (cancer, fraud) → maximise recall. False alarm is costly (spam filter, automated action) → maximise precision.

Sigmoid vs softmax at the output

Multi-class (exactly one label) → softmax, outputs sum to 1. Multi-label (several can be true) → sigmoid, independent probabilities.

Keras loss selection

Xavier vs He

Xavier ↔ tanh/sigmoid. He ↔ ReLU. He uses twice the variance because ReLU zeroes half the activations.

What fixes what

Zero initialisation

Every neuron in the layer computes the same thing, receives the same gradient, and stays identical forever. The layer has the capacity of one neuron. This is the symmetry problem.

Epochs, batches, iterations

iterations per epoch = ceil(n_samples / batch_size). 60,000 samples, batch 128, 20 epochs → 469 × 20 = 9,380 updates.

Conv output size

out = floor((in − k + 2p)/s) + 1

in=28, k=5, p=0, s=1 → 24. With p=2 → 28.

Conv parameter count

k·k·C_in·C_out + C_out. Conv2d(16,32,3) → 3·3·16·32 + 32 = 4,640. Dense layer: in·out + out.

Why √d_k in attention

Without it, dot products grow with dimension, softmax saturates to near-one-hot, and gradients vanish. Hence scaled dot-product attention.

LayerNorm vs BatchNorm

BatchNorm — across the batch, per feature. Batch-size dependent, different at train and eval. Used in CNNs. LayerNorm — across features, per sample. Batch-size independent, identical at train and eval. Used in transformers.

The three attention mechanisms

1. Encoder self-attention (bidirectional)
2. Decoder masked self-attention (causal)
3. Cross-attention — decoder Q, encoder K/V. Only exists in encoder-decoder models.

BERT's 80/10/10 split

Of the 15% masked tokens: 80% → [MASK], 10% → random token, 10% → unchanged. Why: [MASK] never appears at fine-tuning time, so training only on it creates a train/inference mismatch. The other two cases force good representations of every token.

Model family selection

GPT / decoder-only → generation, chat, completion BERT / encoder-only → classification, NER, retrieval embeddings T5, BART / encoder-decoder → translation, summarisation

RAG vs fine-tuning

RAG supplies knowledge. Fine-tuning supplies behaviour, style and format. Facts that change → RAG. Consistent output shape or domain tone → fine-tune. Usually both.

Fine-tuning learning rate

2e-5 to 5e-5. Using 1e-3 causes catastrophic forgetting. DPO is lower still: 5e-7.

Why LoRA's B is zero-initialised

So B·A = 0 at step 0 and the model starts exactly as the pretrained model. Random init on both would perturb a good model before training begins.

QLoRA's three innovations

1. NF4 (4-bit NormalFloat)
2. Double quantisation
3. Paged optimizers

PPO's four models

Policy, reference (frozen, for KL), reward model, value model/critic. This is why RLHF is expensive and why DPO — which needs two — took over.

Why the DPO partition function cancels

The pairwise Bradley-Terry loss depends only on the difference between two rewards for the same prompt. β·log Z(x) is identical for both, so it cancels exactly.

CBOW vs skip-gram

CBOW = context → word (faster, better on frequent words). Skip-gram = word → context (better on rare words and small corpora).

Perplexity

exp(cross-entropy). Lower is better. Only comparable across models sharing a tokenizer and test set.

Chain vs agent

Chain — you fix the steps. Predictable, cheap. Agent — the LLM chooses the steps at runtime. Flexible, expensive, can loop.

LangChain chain_type

stuff — all chunks in one prompt (default) map_reduce — per chunk then combine (many chunks, parallel) refine — iteratively improve (quality, slow) map_rerank — score each, return the best (single-chunk answers)

---

These produce no error. They just make your results wrong.

1. fit_transform on the test set

X_test = scaler.fit_transform(X_test)     # WRONG — leaks test statistics
X_test = scaler.transform(X_test)         # right

2. Preprocessing before splitting

Scaling or imputing on the full dataset before train_test_split leaks test information into training. Use a Pipeline. This is the leakage that most often produces a validation score that collapses in production.

3. Forgetting optimizer.zero_grad()

PyTorch accumulates gradients. Omit the reset and every step uses the sum of all previous gradients. Training silently produces garbage.

4. Softmax before CrossEntropyLoss

nn.Sequential(..., nn.Softmax(dim=1))     # WRONG with CrossEntropyLoss
nn.Sequential(..., nn.Linear(h, C))       # right — raw logits

CrossEntropyLoss applies LogSoftmax internally. Double softmax flattens gradients. No error. Same for BCEWithLogitsLoss vs BCELoss.

5. Shape broadcasting in the loss

preds   # (100, 1)
targets # (100,)
loss = criterion(preds, targets)     # broadcasts to (100,100). No error. Completely wrong.
targets = targets.view(-1, 1)        # fix

6. Missing model.eval()

Dropout stays active and BatchNorm uses batch statistics at inference. Predictions become random and inconsistent. The classic "works in training, garbage at test time".

Its mirror image: forgetting model.train() afterwards, so dropout never activates.

7. Plain Python list of layers

self.layers = [nn.Linear(10,10) for _ in range(3)]     # NOT registered
self.layers = nn.ModuleList([...])                      # registered

With a plain list, model.parameters() returns nothing, the optimiser has nothing to update, and training does absolutely nothing.

8. Not recompiling after changing trainable (Keras)

base.trainable = True
model.compile(...)     # REQUIRED — otherwise the change has no effect

9. Fine-tuning at the original learning rate

Unfreezing a pretrained base and continuing at 1e-3 destroys the pretrained features. Drop to 1e-5.

10. Mismatched tokenizer and model

A BERT tokenizer with a GPT-2 model produces fluent nonsense. Always load both from the same checkpoint.

11. Missing attention_mask

Padding tokens participate in attention and corrupt the representation. No error.

12. Different embedding models for queries and documents

Different vector spaces. Retrieval returns results; they're meaningless. Silent.

13. Chunks longer than the embedding model's max input

Silently truncated. A 512-token embedding model with 2000-character chunks discards the tail of every chunk.

14. Augmenting the validation set

Inflates and destabilises validation scores. Augment training only.

15. Channels-first vs channels-last

Keras is HWC; PyTorch is CHW. ToTensor() does the conversion. Mixing them up between courses 3 and 5 produces bizarre shape errors — or worse, no error and a model learning from transposed data.

16. Model and data on different devices

RuntimeError: Expected all tensors to be on the same device

Move both: model.to(device) and x.to(device).

17. No random seed

Run-to-run variance gets mistaken for a real difference between models or frameworks. Set random_state, torch.manual_seed, np.random.seed, tf.random.set_seed.

18. Gradio file object vs path

gr.File(type="filepath")     # required — PDF loaders want a path string

19. Wrong preprocess_input for a pretrained model

Each pretrained vision model expects specific normalisation. The wrong one silently degrades accuracy with no error.

20. Shuffling time-series data

shuffle=True on temporal data trains on the future to predict the past. Excellent scores, worthless model. Use TimeSeriesSplit.

---

Debugging order when a model underperforms

1. Look at the data. Print samples, check labels, check class balance, check for duplicates.
2. Check shapes at every stage.
3. Overfit one batch deliberately. If the model can't drive loss to ~0 on 8 samples,

something is structurally broken — not a tuning problem.

1. Check train() / eval() and zero_grad().
2. Check the loss/activation pairing — double softmax, wrong loss for the label format.
3. Then tune the learning rate. It's the highest-impact hyperparameter, and the first thing

people reach for when it should be the fifth.

For RAG specifically:

1. Print the extracted text — is the loader working?
2. Print the chunks — are they coherent?
3. similarity_search_with_score — was the right chunk retrieved at all?
4. Only then look at the prompt and the model.