Motion graphic

Introduction to Deep Learning & Neural Networks with Keras

10 hours · 5 modules · Keras (TensorFlow backend) Final project: Classification and Captioning.

Shortest course in the certificate, and the most conceptually important. This is where the mechanics of a neural network get explained. Everything after this assumes you have it.

---

Module 1 — Introduction to Deep Learning and Neural Networks

Core ideas

Why deep learning happened when it did — the algorithms are old (backprop is 1986). Three things changed: data volume, GPU compute, and better initialisation/activation tricks that made deep networks actually trainable.

Biological neuron → artificial neuron. The videos draw the analogy: dendrites (inputs), soma (summation), axon (output), synapse strength (weights). Useful as a mnemonic; don't take it literally — an artificial neuron is a weighted sum plus a non-linearity, nothing more.

The artificial neuron:

z = Σ (wᵢ · xᵢ) + b        ← linear part: weighted sum plus bias
a = f(z)                    ← non-linear activation

The bias b shifts the activation threshold — without it, every neuron's decision boundary would be forced through the origin.

Network anatomy:

• Input layer — one node per feature. Does no computation.
• Hidden layers — where representation-learning happens
• Output layer — shape and activation determined by the task
• Weights and biases — the learned parameters
• A layer is dense / fully-connected when every node connects to every node in the next layer

Why non-linearity is non-negotiable: stack two linear layers and you get W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂) — which is just another linear layer. Without an activation function, a 100-layer network has exactly the expressive power of a 1-layer one. This is a favourite exam question.

Forward propagation

Data flows input → output, one layer at a time. For layer l:

z⁽ˡ⁾ = W⁽ˡ⁾ · a⁽ˡ⁻¹⁾ + b⁽ˡ⁾
a⁽ˡ⁾ = f(z⁽ˡ⁾)

That's it — a matrix multiply, an add, an element-wise function, repeated per layer. The Module 1 lab has you code this by hand in NumPy. Do it properly; understanding shapes here saves you hours of debugging later.

Shape discipline. If x is (features, 1) and layer 1 has 5 nodes, W⁽¹⁾ is (5, features) and b⁽¹⁾ is (5, 1). Most beginner errors in the lab are transposition errors.

---

Module 2 — Basics of Deep Learning

Gradient descent

The learning algorithm. You have a loss J(w); you want the w that minimises it. The gradient ∂J/∂w points uphill, so step downhill:

w := w − α · ∂J/∂w

α (learning rate) is the step size, and it's the hyperparameter that matters most:

• Too small → training crawls, may stall in a flat region
• Too large → overshoots, loss oscillates or diverges to NaN
• Typical starting points: 0.01, 0.001, 0.0001

Gradient descent finds a local minimum. In high-dimensional networks this turns out to be fine in practice — most local minima are about as good as each other, and the real obstacles are saddle points and plateaus, not bad minima.

Backpropagation

How you get ∂J/∂w for every weight in the network. It's the chain rule, applied backwards through the layers so intermediate results get reused instead of recomputed.

The four-step training loop, which is the whole of deep learning:

1. Forward pass — compute predictions
2. Compute loss — compare predictions to labels
3. Backward pass — propagate the error gradient back, layer by layer
4. Update — nudge every weight against its gradient

Repeat for every batch, for every epoch.

Chain rule, spelled out for one weight:

∂J/∂w⁽ˡ⁾ = (∂J/∂a⁽ˡ⁾) · (∂a⁽ˡ⁾/∂z⁽ˡ⁾) · (∂z⁽ˡ⁾/∂w⁽ˡ⁾)
             ↑ from the      ↑ derivative     ↑ = a⁽ˡ⁻¹⁾
             layer above     of activation

Note the middle term: the derivative of the activation function multiplies into every gradient. That single fact explains the vanishing gradient problem.

Vanishing gradients

Sigmoid squashes everything into (0, 1), and its derivative peaks at 0.25.

Multiply 0.25 by itself through many layers: 0.25⁵ ≈ 0.001, 0.25¹⁰ ≈ 0.0000009. The gradient reaching the early layers is effectively zero, so those layers stop learning. The network trains its last few layers and leaves the rest at their random initialisation.

tanh is a bit better (derivative peaks at 1.0) but still saturates at the tails.

The fix: ReLU. Its derivative is exactly 1 for all positive inputs — no shrinkage, no matter how deep. This is the single change that made deep networks trainable, and it's the answer to "why did deep learning suddenly work".

Activation functions

Dying ReLU: a neuron whose weights push it permanently negative outputs 0 forever, gets gradient 0 forever, and is dead. Leaky ReLU keeps a small negative slope so there's always some gradient.

Softmax vs sigmoid at the output — the distinction they quiz:

• Sigmoid: independent probabilities. Binary classification, or multi-label (an image

can be both "beach" and "sunset").

• Softmax: a probability distribution over mutually exclusive classes; outputs sum to 1.

Multi-class (a digit is exactly one of 0–9).

---

Module 3 — Keras and Deep Learning Libraries

The library landscape

• TensorFlow — Google. Production, deployment, mobile/edge (TF Lite), serving.
• Keras — the high-level API, now built into TensorFlow as tf.keras. Fast to write.
• PyTorch — Meta. Dominant in research; dynamic graphs; what courses 4–13 use.
• Theano — historical, dead. Mentioned only for context.

The certificate teaches both Keras and PyTorch on purpose: Keras for speed of expression, PyTorch because that's what the GenAI ecosystem (Hugging Face) is built on.

Regression with Keras

import keras
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(50, activation='relu', input_shape=(n_cols,)))
model.add(Dense(50, activation='relu'))
model.add(Dense(1))                              # no activation → raw number

model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_train, y_train, validation_split=0.2, epochs=100, verbose=2)

preds = model.predict(X_test)

Three things define the model shape:

• Output layer size — 1 for regression
• Output activation — none (linear) for regression
• Loss — MSE for regression

Classification with Keras

from keras.utils import to_categorical

y_train = to_categorical(y_train)                # one-hot encode labels

model = Sequential()
model.add(Dense(50, activation='relu', input_shape=(n_cols,)))
model.add(Dense(50, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10, validation_split=0.2)

The output-layer decision table

Learn this. It answers a large fraction of every Keras quiz in the certificate:

categorical_ vs sparse_categorical_ is purely about label format: one-hot [0,0,1] vs integer 2. Mixing them up is the single most common Keras error in these labs.

Optimisers

• SGD — plain gradient descent, one batch at a time
• SGD + momentum — accumulates a velocity vector; rolls through small bumps and dampens

oscillation

• RMSprop — per-parameter adaptive learning rate based on recent gradient magnitude
• Adam — momentum + RMSprop combined. Use this unless you have a reason not to.

Useful fit arguments

model.fit(X, y,
          epochs=50,             # full passes over the data
          batch_size=32,         # samples per gradient update
          validation_split=0.2,  # hold out 20% for validation
          verbose=2,
          callbacks=[keras.callbacks.EarlyStopping(patience=5,
                                                   restore_best_weights=True)])

Epoch vs batch vs iteration — quizzed:

• 1000 samples, batch_size 100 → 10 iterations = 1 epoch
• Iterations per epoch = ceil(n_samples / batch_size)

EarlyStopping watches validation loss and stops when it stops improving. Cheapest possible defence against overfitting; restore_best_weights=True gives you the best epoch, not the last.

---

Module 4 — Deep Learning Models

Survey module. Each architecture gets a few minutes; you get depth later in courses 3 and 5. What matters here is knowing which architecture for which data shape.

Shallow vs deep

Shallow = one hidden layer. Can theoretically approximate any function (universal approximation theorem) but may need an impractically wide layer. Deep = many hidden layers, and each layer builds features on the previous layer's features. This hierarchy is the point — edges → shapes → parts → objects.

Convolutional Neural Networks (CNN)

For grid-like data: images, video, spectrograms.

• Convolutional layer — slides a small learned filter (kernel) across the input, producing a

feature map. The same weights are used at every position (parameter sharing), which is why a CNN needs vastly fewer parameters than a dense layer on the same image.

• Pooling layer — downsamples (usually max-pool 2×2), giving translation tolerance and

cutting compute.

• Flatten → Dense — final classification head.

Two properties that make CNNs work on images: local receptive fields (nearby pixels are related) and translation invariance (a cat is a cat wherever it is in the frame).

Output size of a conv layer:

out = floor((in − kernel + 2·padding) / stride) + 1

from keras.layers import Conv2D, MaxPooling2D, Flatten

model.add(Conv2D(16, (5,5), activation='relu', input_shape=(28,28,1)))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(32, (3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(100, activation='relu'))
model.add(Dense(10, activation='softmax'))

Keras image input shape is (height, width, channels) — channels last. PyTorch puts channels first. This bites everyone who switches between courses 3 and 5.

Recurrent Neural Networks (RNN)

For sequential data where order matters: text, time series, audio.

A hidden state is carried from timestep to timestep — the network has a memory. But long chains of multiplication cause vanishing gradients, so plain RNNs can't learn long-range dependencies.

LSTM (Long Short-Term Memory) fixes this with a cell state and three gates — forget, input, output — that control what's kept, added and emitted. GRU is a lighter variant with two gates.

Transformers

Process the whole sequence in parallel rather than step by step, using self-attention so every token can look directly at every other token. No recurrence, no vanishing gradient over sequence length, and it parallelises on GPUs — which is why transformers displaced RNNs and made today's LLMs possible.

Courses 9–11 cover this properly. Here just fix the comparison:

Autoencoders

Unsupervised. Encoder compresses input to a low-dimensional bottleneck (latent representation); decoder reconstructs the input from it. Trained to make output ≈ input, so the labels are the inputs themselves.

Uses: dimensionality reduction (non-linear PCA), denoising, anomaly detection (high reconstruction error = anomaly).

Restricted Boltzmann Machines get a mention — historical, largely superseded.

Using pre-trained models

The idea the entire second half of the certificate rests on: someone else spent millions of GPU-hours learning general features; you reuse them.

from keras.applications import VGG16
base = VGG16(weights='imagenet', include_top=False, input_shape=(224,224,3))
base.trainable = False          # freeze the learned features
# add your own small head on top and train only that

Why it works: early layers learn generic features (edges, textures) that transfer across domains. Only the last layers are task-specific.

---

Module 5 — Final Project and Course Wrap-Up

Final Project: Classification and Captioning — combines a CNN image classifier with a caption-generation component. Peer/AI graded.

What the rubric wants:

1. Data loaded and preprocessed (resized, normalised to [0,1], correctly shaped)
2. A working model with justified architecture choices
3. compile with the loss appropriate to the task
4. Training history plotted — loss and accuracy, train vs validation
5. Evaluation on held-out data
6. Written interpretation, including what you'd change

Standard training-history plot the graders expect:

history = model.fit(X_train, y_train, validation_split=0.2, epochs=20)

import matplotlib.pyplot as plt
plt.plot(history.history['loss'],     label='train')
plt.plot(history.history['val_loss'], label='validation')
plt.xlabel('epoch'); plt.ylabel('loss'); plt.legend()

Reading that plot is a skill they assess:

• Both curves falling → still learning, train longer
• Train falling, validation rising → overfitting, stop earlier / regularise / get more data
• Both flat and high → underfitting, bigger model or higher learning rate
• Validation jagged → learning rate too high, or batch size too small

---

Course 2 in one page

• Neuron = weighted sum + bias + non-linear activation
• No activation → the whole network collapses to one linear layer
• Training loop: forward → loss → backward (chain rule) → update
• Vanishing gradients come from activation derivatives < 1 multiplied through depth; ReLU fixes it
• ReLU for hidden layers; sigmoid for binary output; softmax for multi-class output
• Adam unless you have a reason otherwise
• Loss table: MSE / binary_crossentropy / categorical_crossentropy / sparse_categorical_crossentropy
• CNN → grids. RNN/LSTM → sequences. Transformer → sequences, in parallel. Autoencoder → unsupervised compression
• Keras is channels-last; PyTorch is channels-first
• Pre-trained + frozen base + small custom head = transfer learning, the backbone of everything later