Motion graphic

Introduction to Neural Networks and PyTorch

19 hours · 7 modules · PyTorch Practice: Breast Cancer Classification NN. Final: League of Legends Match Predictor.

The concepts here duplicate Course 1 and 2 (regression, gradient descent, logistic regression). What's new is PyTorch itself — and PyTorch is what every course from 7 onward uses, because Hugging Face is built on it. Learn the idioms; skim the theory you already have.

---

Module 1 — Exploring Tensors

Tensors

A tensor is an n-dimensional array with two extras NumPy doesn't have: it can live on a GPU, and it can track its own gradients.

0-D scalar · 1-D vector · 2-D matrix · 3-D+ tensor

import torch

a = torch.tensor([1, 2, 3])                      # from a list
b = torch.tensor([[1., 2.], [3., 4.]])           # 2-D
z = torch.zeros(3, 4); o = torch.ones(2, 2)
r = torch.randn(3, 3)                            # standard normal
e = torch.arange(0, 10, 2)                       # 0,2,4,6,8
l = torch.linspace(0, 1, steps=5)                # 5 points, 0 to 1 inclusive

a.dtype      # torch.int64
a.shape      # torch.Size([3])
a.ndimension()
a.size()

dtype matters. torch.tensor([1,2,3]) is int64; feed that to a float layer and you get a type error. Default float type is torch.float32. Convert with .float(), .long(), .type(torch.FloatTensor).

1-D operations

u = torch.tensor([1., 2.]); v = torch.tensor([3., 4.])

u + v                 # element-wise add
u * v                 # element-wise (Hadamard) product — NOT matrix multiply
torch.dot(u, v)       # dot product → scalar (11.)
u + 1                 # broadcasting: scalar applied to every element
torch.mean(u); torch.max(u); torch.std(u)
torch.sin(u)

u[0]                  # index
u[1:3]                # slice

Reshaping and views

a = torch.arange(12)
a.view(3, 4)          # reshape; requires contiguous memory, shares storage
a.view(-1, 4)         # -1 = infer this dimension
a.reshape(3, 4)       # like view but copies if it must
a.unsqueeze(0)        # add a dimension → shape (1, 12)
a.squeeze()           # remove size-1 dimensions
a.t()                 # transpose (2-D)

view shares memory with the original — mutating one mutates the other. reshape may copy. When in doubt use reshape.

NumPy and pandas interop

np_arr = tensor.numpy()               # tensor → numpy (SHARES memory)
tensor = torch.from_numpy(np_arr)     # numpy → tensor (SHARES memory)
tensor = torch.tensor(df.values)      # from a DataFrame
tensor.item()                         # single-element tensor → Python scalar

Shared memory is a real trap: change the NumPy array in place and the tensor changes too. .clone() to break the link.

2-D tensors

A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[5., 6.], [7., 8.]])

A * B                 # element-wise
torch.mm(A, B)        # matrix multiply (2-D only)
A @ B                 # same, and works for batched/n-D
torch.matmul(A, B)    # general, with broadcasting

A[0, 1]               # row 0, col 1
A[:, 0]               # column 0
A.shape               # torch.Size([2, 2])

Matrix multiply rule: (m×n) @ (n×p) → (m×p). Inner dimensions must match. Roughly 80% of PyTorch runtime errors are shape mismatches here — read the error, it tells you both shapes.

Differentiation — autograd

The feature that makes PyTorch a deep learning framework rather than NumPy-on-GPU.

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 2 * x + 1
y.backward()                 # compute dy/dx
print(x.grad)                # 2x + 2 = 6.0

• requires_grad=True marks a tensor to be tracked
• Every op on it builds a dynamic computation graph
• .backward() walks the graph backwards, accumulating gradients into .grad

Partial derivatives:

u = torch.tensor(1.0, requires_grad=True)
v = torch.tensor(2.0, requires_grad=True)
f = u * v + u ** 2
f.backward()
u.grad     # v + 2u = 4
v.grad     # u = 1

Three rules that cause real bugs:

1. Gradients accumulate. .backward() adds to .grad. You must call

optimizer.zero_grad() (or x.grad.zero_()) every iteration or gradients from every previous step pile up. Forgetting this is the classic PyTorch bug — training silently produces garbage.

1. The graph is freed after .backward(). Calling it twice on the same graph errors;

use retain_graph=True if you genuinely need to.

1. Wrap inference in torch.no_grad() — otherwise you build a graph you'll never use and

waste memory.

with torch.no_grad():
    preds = model(X_test)

Dynamic vs static graphs — PyTorch builds the graph as the code runs, so you can use normal Python control flow and print() mid-forward-pass. That "define-by-run" property is why research moved to PyTorch.

---

Module 2 — Building Datasets in PyTorch

The Dataset class

Three required pieces:

from torch.utils.data import Dataset

class MyDataset(Dataset):
    def __init__(self, X, y, transform=None):
        self.x = torch.tensor(X, dtype=torch.float32)
        self.y = torch.tensor(y, dtype=torch.float32)
        self.len = self.x.shape[0]
        self.transform = transform

    def __getitem__(self, index):            # return ONE sample
        sample = self.x[index], self.y[index]
        if self.transform:
            sample = self.transform(sample)
        return sample

    def __len__(self):                       # how many samples
        return self.len

__getitem__ returns a single sample; __len__ returns the count. That's the whole contract — everything else in the PyTorch data stack builds on those two methods.

DataLoader

Wraps a Dataset and handles batching, shuffling and parallel loading:

from torch.utils.data import DataLoader

loader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=2)

for x_batch, y_batch in loader:
    ...

• shuffle=True for training, False for validation/test
• num_workers — parallel loading processes. On Windows, set 0 or guard your script with

if __name__ == '__main__': or it will fork-bomb.

• drop_last=True discards a final undersized batch (matters for BatchNorm)

Image datasets and transforms

import torchvision.transforms as transforms
from torchvision import datasets

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),                    # PIL/numpy → tensor, scales to [0,1], CHW order
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),   # ImageNet stats
])

train = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
# or from a folder tree where each subdirectory is a class:
train = datasets.ImageFolder(root='data/train', transform=transform)

Two things to burn in:

• ToTensor() converts HWC → CHW and scales to [0,1]. PyTorch is channels-first,

Keras is channels-last. This is the most common cross-framework confusion in the certificate.

• Normalize must come after ToTensor() — it operates on tensors.
• Those ImageNet mean/std numbers are what every pretrained torchvision model expects.

---

Module 3 — Applying Linear Regression and Gradient Descent

Prediction, three ways

Manual:

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)
def forward(x): return w * x + b

With nn.Linear:

from torch import nn
model = nn.Linear(in_features=1, out_features=1)
yhat = model(torch.tensor([[1.0]]))
list(model.parameters())        # weight and bias, auto-initialised

Custom module — the pattern you'll use for the rest of the certificate:

class LR(nn.Module):
    def __init__(self, in_size, out_size):
        super().__init__()                       # ALWAYS call this first
        self.linear = nn.Linear(in_size, out_size)

    def forward(self, x):
        return self.linear(x)

model = LR(1, 1)
yhat  = model(x)          # calls forward() — never call model.forward(x) directly

Call model(x), not model.forward(x) — the __call__ wrapper runs registered hooks that forward() alone skips.

Loss and cost

Loss = error on one sample. Cost = average loss over the dataset. The videos are strict about this distinction and the quiz tests it.

MSE cost:  J(w,b) = (1/N) · Σ (yᵢ − ŷᵢ)²

criterion = nn.MSELoss()          # regression
loss = criterion(yhat, y)

Gradient descent by hand

Worth writing once so the abstraction later makes sense:

lr = 0.1
for epoch in range(20):
    yhat = forward(X)
    loss = criterion(yhat, Y)

    loss.backward()                       # compute gradients
    with torch.no_grad():                 # update outside the graph
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad.zero_(); b.grad.zero_()        # RESET

The loss surface for linear regression is a convex bowl — one global minimum, no local traps. That's why they teach gradient descent here first.

Batch / Stochastic / Mini-batch

The noise in SGD is not purely a defect: it helps escape saddle points and acts as a mild regulariser.

---

Module 4 — Training Linear Regression the PyTorch Way

The canonical training loop

This exact structure appears in every remaining course. Memorise it:

from torch import nn, optim
from torch.utils.data import DataLoader

model     = LR(1, 1)
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)
loader    = DataLoader(dataset, batch_size=32, shuffle=True)

for epoch in range(epochs):
    model.train()
    for x, y in loader:
        optimizer.zero_grad()          # 1. clear old gradients
        yhat = model(x)                # 2. forward
        loss = criterion(yhat, y)      # 3. loss
        loss.backward()                # 4. backward
        optimizer.step()               # 5. update

zero_grad → forward → loss → backward → step. Five lines, in that order. Swap or omit one and it either crashes or silently trains wrong.

Validation

    model.eval()                       # dropout off, batchnorm uses running stats
    with torch.no_grad():
        val_loss = sum(criterion(model(xv), yv).item() for xv, yv in val_loader)

model.train() / model.eval() change behaviour of Dropout and BatchNorm layers. Forgetting eval() at inference gives inconsistent predictions; forgetting train() afterwards means dropout never activates. Both are silent failures.

Train / validation / test

• Train — fit parameters
• Validation — choose hyperparameters, decide when to stop
• Test — touched once, at the very end

from torch.utils.data import random_split
train_ds, val_ds = random_split(dataset, [0.8, 0.2])

Typical hyperparameter sweep, and the pattern the lab wants:

best_loss, best_lr = float('inf'), None
for lr in [1e-1, 1e-2, 1e-3, 1e-4]:
    model = LR(1,1); optimizer = optim.SGD(model.parameters(), lr=lr)
    ... train ...
    if val_loss < best_loss:
        best_loss, best_lr, best_model = val_loss, lr, model

Optimisers

optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
optim.Adam(model.parameters(), lr=0.001)
optim.RMSprop(model.parameters(), lr=0.01)

Adam is the safe default. SGD + momentum sometimes generalises slightly better on vision tasks with a tuned schedule.

---

Module 5 — Multiple Inputs and Outputs

Multiple inputs

model = nn.Linear(in_features=3, out_features=1)   # 3 features → 1 prediction

Input tensor shape is (batch_size, 3). Weight matrix is (1, 3). Under the hood: ŷ = Xw + b, where X is (N×3), w is (3×1).

class MLR(nn.Module):
    def __init__(self, in_size, out_size):
        super().__init__()
        self.linear = nn.Linear(in_size, out_size)
    def forward(self, x):
        return self.linear(x)

model = MLR(3, 1)

Multiple outputs

Change out_features. One nn.Linear(3, 2) predicts two targets simultaneously — internally two independent linear functions sharing an input.

model = nn.Linear(3, 2)          # weight shape (2,3), bias shape (2,)

MSELoss handles multi-output automatically by averaging across all elements.

Current trends in PyTorch (mentioned in the videos)

torch.compile() for graph compilation and speed; better distributed training APIs; torch.func for functional transforms; ExecuTorch for on-device; growing dominance of the Hugging Face ecosystem built on top of PyTorch.

---

Module 6 — Logistic Regression for Classification

Linear classifiers and the sigmoid

z = wᵀx + b            ŷ = σ(z) = 1/(1+e^(−z))

Threshold at 0.5 → class 1 when z > 0. The decision boundary is the hyperplane z = 0.

class LogisticRegression(nn.Module):
    def __init__(self, in_size):
        super().__init__()
        self.linear = nn.Linear(in_size, 1)
    def forward(self, x):
        return torch.sigmoid(self.linear(x))

Or, more simply: nn.Sequential(nn.Linear(1,1), nn.Sigmoid()).

Bernoulli and maximum likelihood

The theoretical grounding the videos walk through:

A binary outcome is Bernoulli-distributed: P(y|x) = ŷ^y · (1−ŷ)^(1−y). For independent samples, the likelihood is the product over all samples. Products of small numbers underflow, so take the log (turning products into sums), and negate to get a minimisation problem:

Negative log-likelihood = −Σ [ y·log(ŷ) + (1−y)·log(1−ŷ) ]

That is cross-entropy loss. The point of the derivation: cross-entropy isn't an arbitrary choice — it's maximum likelihood estimation for a Bernoulli output.

Why not MSE

The videos demonstrate this explicitly in a lab, and it's a guaranteed exam question:

• MSE + sigmoid gives a non-convex surface with flat plateaus. When the model is confidently

wrong, the sigmoid saturates, its derivative → 0, and the MSE gradient vanishes — the model learns slowest exactly when it's most wrong.

• Cross-entropy is convex in this setting, and its gradient simplifies to (ŷ − y)·x: the

bigger the error, the bigger the correction. Precisely the behaviour you want.

Run the "Logistic Regression Mean Square Error" and "Logistic Regression Cross Entropy" labs side by side and compare the loss curves — that's the whole lesson.

Loss functions in PyTorch

nn.BCELoss()                 # expects probabilities — apply sigmoid yourself
nn.BCEWithLogitsLoss()       # expects RAW LOGITS — applies sigmoid internally
nn.CrossEntropyLoss()        # multi-class; expects RAW LOGITS + integer class labels
nn.NLLLoss()                 # expects log-probabilities (after LogSoftmax)

The single biggest PyTorch trap in this certificate:

nn.CrossEntropyLoss applies LogSoftmax internally. If you also put a Softmax on your output layer, you are applying softmax twice. It won't error. It will just train badly.

Same for BCEWithLogitsLoss vs BCELoss. Prefer the WithLogits versions — they're numerically stable (log-sum-exp trick) and avoid the double-activation bug.

Also: CrossEntropyLoss wants targets as class indices ([0, 2, 1]), not one-hot vectors. The opposite convention from Keras categorical_crossentropy.

Regularisation in PyTorch

L2 / weight decay — a parameter on the optimiser, not a layer:

optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

Early stopping — no built-in; you write it:

best, patience, wait = float('inf'), 5, 0
for epoch in range(epochs):
    ...
    if val_loss < best:
        best, wait = val_loss, 0
        torch.save(model.state_dict(), 'best.pt')
    else:
        wait += 1
        if wait >= patience:
            break
model.load_state_dict(torch.load('best.pt'))

Saving and loading — always state_dict, never the whole model object (pickling the object couples the file to your exact class definition and file layout):

torch.save(model.state_dict(), 'model.pt')
model.load_state_dict(torch.load('model.pt'))
model.eval()

---

Module 7 — Final Project, Final Quiz, Wrap-Up

Practice project: Neural Network for Breast Cancer Classification (Wisconsin dataset, binary, tabular). Final project: League of Legends Match Predictor — binary classification from match statistics. AI-graded or peer-graded.

Rubric:

1. Data loaded into a Dataset and DataLoader, features scaled
2. nn.Module subclass with a justified architecture
3. Correct loss (BCEWithLogitsLoss or CrossEntropyLoss) and optimiser
4. Full training loop with the five steps, tracking train and validation loss
5. Loss curves plotted
6. Evaluation: accuracy, confusion matrix, precision/recall
7. Model weights saved via state_dict
8. Written interpretation

Full working skeleton:

import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader

X_tr = torch.tensor(X_train, dtype=torch.float32)
y_tr = torch.tensor(y_train, dtype=torch.float32).view(-1, 1)   # note the view

loader = DataLoader(TensorDataset(X_tr, y_tr), batch_size=32, shuffle=True)

model = nn.Sequential(
    nn.Linear(X_tr.shape[1], 64), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(64, 32),            nn.ReLU(),
    nn.Linear(32, 1),                                  # raw logits — no sigmoid
)
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

for epoch in range(100):
    model.train()
    for xb, yb in loader:
        optimizer.zero_grad()
        loss = criterion(model(xb), yb)
        loss.backward()
        optimizer.step()

model.eval()
with torch.no_grad():
    probs = torch.sigmoid(model(X_te))      # sigmoid HERE, at inference
    preds = (probs > 0.5).float()

Note where the sigmoid lives: not in the model, because BCEWithLogitsLoss handles it during training; applied manually at inference when you want probabilities.

Final exam focus

• Tensor shapes and matrix-multiply compatibility
• What requires_grad, backward(), .grad and no_grad() each do
• The five-step training loop, in order
• Which loss for which task, and the logits-vs-probabilities distinction
• Dataset.__getitem__ / __len__
• model.train() vs model.eval()
• Batch vs stochastic vs mini-batch gradient descent

---

Course 4 in one page

• Tensor = array + GPU + autograd
• requires_grad=True → ops recorded → .backward() → read .grad
• Gradients accumulate — zero_grad() every iteration
• Dataset needs __getitem__ and __len__; DataLoader batches and shuffles
• ToTensor() = HWC→CHW, scale to [0,1]; PyTorch is channels-first
• Custom model: subclass nn.Module, super().__init__(), define forward, call model(x)
• zero_grad → forward → loss → backward → step
• model.train() / model.eval() around dropout and batchnorm
• Cross-entropy = negative log-likelihood of a Bernoulli/categorical output; MSE + sigmoid saturates
• CrossEntropyLoss and BCEWithLogitsLoss take raw logits — never softmax before them
• Targets: .view(-1,1) to match prediction shape, or the loss broadcasts silently
• Save state_dict(), not the model object