Motion graphic

Deep Learning with Keras and TensorFlow

23 hours · 7 modules · the longest course in the certificate Practice project: Fruit Classification (transfer learning). Final: Classify Waste Products.

Course 2 taught you Sequential. This course is about everything Sequential can't do.

---

Module 1 — Advanced Keras Functionalities

Three ways to build a Keras model

1. Sequential — a linear stack. Simple, and limited to one input, one output, no branching.

2. Functional API — you wire layers together like a graph. Handles multiple inputs/outputs, shared layers, skip connections, branching.

from keras.layers import Input, Dense, concatenate
from keras.models import Model

inp    = Input(shape=(20,))
x      = Dense(64, activation='relu')(inp)
x      = Dense(64, activation='relu')(x)
out    = Dense(1,  activation='sigmoid')(x)
model  = Model(inputs=inp, outputs=out)

The pattern is layer(tensor) → tensor: each layer is called on the previous output. A layer object is callable, and calling it wires it into the graph.

Multi-input / multi-output — the thing Sequential simply cannot express:

in_a = Input(shape=(32,), name='numeric')
in_b = Input(shape=(128,), name='text_embedding')

a = Dense(16, activation='relu')(in_a)
b = Dense(16, activation='relu')(in_b)
merged = concatenate([a, b])

out_main = Dense(1, activation='sigmoid', name='main')(merged)
out_aux  = Dense(1, activation='sigmoid', name='aux')(a)

model = Model(inputs=[in_a, in_b], outputs=[out_main, out_aux])
model.compile(optimizer='adam',
              loss={'main': 'binary_crossentropy', 'aux': 'binary_crossentropy'},
              loss_weights={'main': 1.0, 'aux': 0.3})

3. Subclassing — write a Python class. Maximum flexibility, full imperative control flow (if statements, loops that depend on the data). Harder to debug and to serialise.

class MyModel(keras.Model):
    def __init__(self, num_classes):
        super().__init__()
        self.d1  = Dense(64, activation='relu')
        self.d2  = Dense(64, activation='relu')
        self.out = Dense(num_classes, activation='softmax')

    def call(self, inputs, training=False):
        x = self.d1(inputs)
        x = self.d2(x)
        return self.out(x)

When to use which — the quiz question:

Custom layers

Subclass keras.layers.Layer. Three methods:

class MyDense(keras.layers.Layer):
    def __init__(self, units, **kwargs):
        super().__init__(**kwargs)
        self.units = units

    def build(self, input_shape):          # called once, when shapes are known
        self.w = self.add_weight(shape=(input_shape[-1], self.units),
                                 initializer='glorot_uniform', trainable=True)
        self.b = self.add_weight(shape=(self.units,),
                                 initializer='zeros', trainable=True)

    def call(self, inputs):                # the forward pass
        return tf.nn.relu(tf.matmul(inputs, self.w) + self.b)

• __init__ — store configuration
• build — create weights (deferred until the input shape is known — that's why you don't have

to declare input dimensions)

• call — define the computation

Use add_weight, not raw tf.Variable, or Keras won't track the parameter and it won't train.

TensorFlow 2.x

• Eager execution by default — operations run immediately, like NumPy. Debuggable with

print(). (TF 1.x built a static graph first, which was the main reason people preferred PyTorch.)

• @tf.function — decorate a function to compile it into a graph for speed. Best of both:

write eagerly, deploy compiled.

• tf.keras is the official high-level API
• tf.data — input pipelines: .batch(), .shuffle(), .prefetch(), .map()
• tf.GradientTape — records operations so you can compute gradients manually (needed in

Module 5's custom training loops)

---

Module 2 — Advanced CNNs in Keras

Data augmentation

Artificially expand the training set with label-preserving transformations. The cheapest regularisation there is, and the standard fix for small image datasets.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=20, width_shift_range=0.2, height_shift_range=0.2,
    shear_range=0.15, zoom_range=0.2, horizontal_flip=True,
    fill_mode='nearest', rescale=1./255,
)
train_gen = datagen.flow_from_directory('data/train', target_size=(224,224),
                                        batch_size=32, class_mode='categorical')

Modern equivalent — augmentation layers inside the model, so they run on GPU and are automatically disabled at inference:

augment = keras.Sequential([
    keras.layers.RandomFlip('horizontal'),
    keras.layers.RandomRotation(0.1),
    keras.layers.RandomZoom(0.1),
])

Augment training data only — never validation or test. And choose transforms that preserve the label: horizontal flip is fine for cats, catastrophic for handwritten digits (6 ↔ 9-ish) and for text.

Transfer learning

The core technique of the module.

base = keras.applications.VGG16(weights='imagenet', include_top=False,
                                input_shape=(224,224,3))
base.trainable = False                                   # freeze

model = keras.Sequential([
    base,
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(256, activation='relu'),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(num_classes, activation='softmax'),
])
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss='categorical_crossentropy',
              metrics=['accuracy'])

Two-stage recipe — this is what the labs and the final project want:

1. Feature extraction — freeze the whole base, train only your new head, normal learning

rate (1e-3). The head starts random; if you don't do this first, its huge random gradients would wreck the pretrained weights.

1. Fine-tuning — unfreeze the top few base layers, retrain with a much smaller learning

rate (1e-5).

base.trainable = True
for layer in base.layers[:-4]:
    layer.trainable = False
model.compile(optimizer=keras.optimizers.Adam(1e-5),   # MUST recompile after changing trainable
              loss='categorical_crossentropy', metrics=['accuracy'])

Two things people get wrong: forgetting to recompile after flipping trainable (the change doesn't take effect otherwise), and fine-tuning at the original learning rate (which destroys the pretrained features).

Choosing how much to unfreeze:

Pretrained models available: VGG16/19, ResNet50, InceptionV3, MobileNetV2 (small/mobile), EfficientNet, Xception. Each has a matching preprocess_input — use it; they expect different input normalisation and using the wrong one silently degrades accuracy.

TensorFlow for image processing

tf.image ops: resize, random_flip_left_right, adjust_brightness, adjust_contrast, central_crop, rgb_to_grayscale. Use inside a tf.data pipeline via .map().

Transpose convolution

Also called deconvolution (a misnomer) or fractionally-strided convolution. It upsamples — it takes a small feature map and produces a larger one, learning the interpolation.

Where it's used: autoencoder decoders, GAN generators, semantic segmentation (U-Net), super-resolution.

keras.layers.Conv2DTranspose(filters=64, kernel_size=(3,3), strides=(2,2), padding='same')

Output size:

out = (in − 1) · stride − 2·padding + kernel

Note it's the convolution formula rearranged — transpose convolution reverses the shape transformation of a convolution, not the values.

Checkerboard artifacts — when kernel_size isn't divisible by stride, output pixels get uneven numbers of contributions and you get a visible grid pattern. Fix: make kernel divisible by stride (4×4 with stride 2), or use UpSampling2D + a normal Conv2D instead.

---

Module 3 — Transformers in Keras

Full theory is Course 9. Here it's implementation.

Why transformers for sequences

Self-attention lets every position attend to every other position directly — one hop instead of n sequential steps. Parallelisable, and no gradient decay over sequence length.

Cost: attention is O(n²) in sequence length. Doubling context quadruples compute. This is the constraint behind every "context window" discussion in the LLM half of the certificate.

Positional encoding

Attention is permutation-invariant — it has no idea what order the tokens are in. So you explicitly add position information to each embedding:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Sinusoids at geometrically-spaced frequencies: fast-varying dimensions encode fine position, slow-varying encode coarse. It extrapolates to sequence lengths not seen in training, and relative positions are expressible as linear functions of the encoding.

Alternative: a learned positional embedding (a lookup table). Simpler, works as well in-distribution, doesn't extrapolate beyond trained length.

Building blocks in Keras

from keras.layers import MultiHeadAttention, LayerNormalization, Dense, Dropout

class TransformerBlock(keras.layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):
        super().__init__()
        self.att  = MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)
        self.ffn  = keras.Sequential([Dense(ff_dim, activation='relu'), Dense(embed_dim)])
        self.ln1  = LayerNormalization(epsilon=1e-6)
        self.ln2  = LayerNormalization(epsilon=1e-6)
        self.do1  = Dropout(rate)
        self.do2  = Dropout(rate)

    def call(self, inputs, training=False):
        attn = self.att(inputs, inputs)                 # self-attention: Q=K=V=inputs
        attn = self.do1(attn, training=training)
        out1 = self.ln1(inputs + attn)                  # residual + norm
        ffn  = self.ffn(out1)
        ffn  = self.do2(ffn, training=training)
        return self.ln2(out1 + ffn)                     # residual + norm

Every transformer block is this shape: attention → add & norm → feed-forward → add & norm. The residual connections (inputs + attn) are what let you stack dozens of these without gradients dying.

LayerNorm vs BatchNorm — quizzed:

• BatchNorm normalises across the batch dimension, per feature. Depends on batch size;

behaves differently at train vs inference. Standard in CNNs.

• LayerNorm normalises across the feature dimension, per sample. Independent of batch

size, identical at train and inference. Standard in transformers, and required for variable sequence lengths.

Transformers for time series

Same block, different framing: treat timesteps as sequence positions, features as embedding dimensions. Add positional encoding, apply a causal mask so a position can't attend to the future, and pool or take the last position for the prediction head.

For forecasting, causal masking is not optional — without it you have leakage from the future, and results that look brilliant and are worthless.

---

Module 4 — Unsupervised Learning and Generative Models in Keras

Autoencoders

input_img = Input(shape=(784,))
encoded   = Dense(64, activation='relu')(input_img)       # bottleneck
decoded   = Dense(784, activation='sigmoid')(encoded)

autoencoder = Model(input_img, decoded)
encoder     = Model(input_img, encoded)                   # reusable feature extractor
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(X_train, X_train, epochs=50)              # target IS the input

fit(X, X) — the label is the input. That's what makes it unsupervised.

Variants:

• Denoising AE — feed corrupted input, target the clean original. Forces genuinely robust

features. fit(X_noisy, X_clean).

• Sparse AE — activity regularisation on the bottleneck
• Convolutional AE — Conv2D encoder, Conv2DTranspose decoder. For images, always this.
• Variational AE (VAE) — the bottleneck encodes a distribution (μ, σ) rather than a point.

Sample from it to generate new data. Loss = reconstruction + KL divergence. This is what makes a VAE generative, whereas a plain autoencoder is only compressive.

Anomaly detection: train on normal data only, then flag anything with high reconstruction error. The model never learned to reconstruct anomalies, so it can't.

GANs

Two networks in a zero-sum game:

• Generator — noise vector z → fake sample. Wants the discriminator to be fooled.
• Discriminator — sample → real/fake probability. Wants to catch fakes.

They train alternately: freeze one, update the other.

# 1. train D on a batch of real (label 1) and a batch of generated (label 0)
d_loss_real = discriminator.train_on_batch(real_images, ones)
d_loss_fake = discriminator.train_on_batch(generator.predict(noise), zeros)

# 2. train G through the frozen combined model, with fake images labelled REAL
discriminator.trainable = False
g_loss = combined.train_on_batch(noise, ones)     # ones = "fool the discriminator"

Failure modes — named in the videos and on the quiz:

• Mode collapse — generator finds one output that reliably fools D and produces only that
• Non-convergence — the two losses oscillate forever instead of settling
• Vanishing gradient — D gets too good too fast, G gets no useful signal

Mitigations: label smoothing, batch norm in G, LeakyReLU in D, Wasserstein loss (WGAN-GP), balancing update frequencies.

Variants mentioned: DCGAN (convolutional), Conditional GAN (class-controlled output), CycleGAN (unpaired image-to-image translation), StyleGAN.

Diffusion models

The architecture behind current image generation (Stable Diffusion, DALL·E, Midjourney), and the one you'll recognise professionally.

Forward process — progressively add Gaussian noise to a real image over T steps until it's pure noise. This is fixed, not learned.

Reverse process — train a network (U-Net) to predict the noise that was added at each step. To generate, start from pure noise and iteratively subtract predicted noise.

Why they beat GANs: stable training (it's a plain regression objective, no adversarial game), better mode coverage, higher fidelity. Cost: generation needs many sequential denoising steps, so it's slow compared to a GAN's single forward pass.

---

Module 5 — Advanced Keras Techniques

Custom training loops

When model.fit() isn't enough — multiple optimisers, custom gradient manipulation, GAN training, non-standard losses.

optimizer = keras.optimizers.Adam(1e-3)
loss_fn   = keras.losses.SparseCategoricalCrossentropy()
metric    = keras.metrics.SparseCategoricalAccuracy()

for epoch in range(epochs):
    for x_batch, y_batch in train_dataset:
        with tf.GradientTape() as tape:
            logits = model(x_batch, training=True)      # forward, recorded
            loss   = loss_fn(y_batch, logits)
        grads = tape.gradient(loss, model.trainable_weights)
        optimizer.apply_gradients(zip(grads, model.trainable_weights))
        metric.update_state(y_batch, logits)
    print(epoch, metric.result().numpy())
    metric.reset_state()

Points that get quizzed:

• Only operations inside the GradientTape context are recorded
• training=True matters — it switches dropout and batch-norm into training behaviour
• A tape is single-use unless created with persistent=True
• Reset metrics each epoch or they accumulate across the whole run

Middle ground: override train_step() in a subclassed model and keep using fit() with all its callbacks. Best of both.

Hyperparameter tuning with Keras Tuner

import keras_tuner as kt

def build_model(hp):
    model = keras.Sequential()
    for i in range(hp.Int('n_layers', 1, 3)):
        model.add(keras.layers.Dense(
            units=hp.Int(f'units_{i}', 32, 512, step=32),
            activation=hp.Choice('activation', ['relu', 'tanh'])))
        model.add(keras.layers.Dropout(hp.Float('dropout', 0.0, 0.5, step=0.1)))
    model.add(keras.layers.Dense(10, activation='softmax'))
    model.compile(
        optimizer=keras.optimizers.Adam(hp.Choice('lr', [1e-2, 1e-3, 1e-4])),
        loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    return model

tuner = kt.Hyperband(build_model, objective='val_accuracy',
                     max_epochs=10, factor=3, directory='tuning', project_name='x')
tuner.search(X_train, y_train, validation_split=0.2)
best = tuner.get_best_models(1)[0]

Search strategies:

• RandomSearch — random combinations. Surprisingly strong baseline.
• Hyperband — trains many configs briefly, kills the losers early, gives survivors more

budget. Best default.

• BayesianOptimization — models the objective surface and picks promising points. Best when

each trial is expensive.

Grid search is exponential in the number of hyperparameters — avoid it for networks.

Model optimisation

Making models smaller and faster for deployment:

• Quantisation — store weights as int8 instead of float32. ~4× smaller, faster on supported

hardware, small accuracy cost. Post-training quantisation is easy; quantisation-aware training recovers more accuracy.

• Pruning — zero out low-magnitude weights. Sparse models compress well.
• Weight clustering — group weights to shared values.
• Knowledge distillation — train a small "student" to mimic a large "teacher's" soft output

probabilities. Often better than training the small model directly on hard labels.

• Mixed precision — float16 compute with float32 master weights. ~2× speedup on modern GPUs.

keras.mixed_precision.set_global_policy('mixed_float16')

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]     # post-training quantisation
tflite_model = converter.convert()

This module is why the certificate lists "Model Optimization" as a skill — remember quantisation and distillation by name for interviews.

---

Module 6 — Introduction to Reinforcement Learning with Keras

Framing

Agent takes an action in an environment, receives a reward and a new state. Goal: learn a policy π(s) → a that maximises cumulative discounted reward.

G_t = R_{t+1} + γ·R_{t+2} + γ²·R_{t+3} + …

γ (discount factor), 0–1: how much future reward is worth now. γ→0 myopic, γ→1 far-sighted.

Distinct from supervised learning in three ways: no labels (only rewards), rewards can be delayed and sparse, and the agent's own actions determine what data it sees.

Exploration vs exploitation — exploit what you know works, or explore to find something better. ε-greedy: with probability ε act randomly, otherwise act greedily. Decay ε over training.

Q-learning

Q(s, a) = expected total future reward from taking action a in state s, then acting optimally. Learn it with the Bellman update:

Q(s,a) ← Q(s,a) + α · [ r + γ · max_a' Q(s',a') − Q(s,a) ]
                        └──────── TD target ────────┘

The bracketed term is the temporal-difference error: what you now think it's worth, minus what you thought before.

Tabular Q-learning stores a table of size |states| × |actions|. Fine for gridworlds; impossible for images (state space is astronomically large).

Q-learning is off-policy — it learns about the greedy policy while behaving ε-greedily. SARSA is the on-policy counterpart. Expect one question on this distinction.

Deep Q-Networks (DQN)

Replace the table with a neural network approximating Q(s, ·).

model = keras.Sequential([
    Dense(24, activation='relu', input_shape=(state_size,)),
    Dense(24, activation='relu'),
    Dense(action_size, activation='linear'),     # one Q-value per action, LINEAR output
])
model.compile(optimizer=Adam(1e-3), loss='mse')

Output activation is linear — Q-values are unbounded returns, not probabilities.

Two stabilising tricks, both examinable:

1. Experience replay — store (s, a, r, s', done) in a buffer, train on random minibatches.

Breaks the temporal correlation between consecutive samples, which otherwise violates the i.i.d. assumption SGD relies on. Also reuses each experience many times.

1. Target network — a periodically-updated frozen copy used to compute the TD target.

Without it you're chasing a target that moves every time you update, and training diverges.

target = reward if done else reward + gamma * np.amax(target_model.predict(next_state)[0])
q_values = model.predict(state)
q_values[0][action] = target
model.fit(state, q_values, epochs=1, verbose=0)

Extensions named in passing: Double DQN (decouples action selection from evaluation, fixes overestimation bias), Dueling DQN, Prioritised Experience Replay.

---

Module 7 — Final Project Submission

Practice: Fruit Classification Using Transfer Learning. Final: Classify Waste Products Using Transfer Learning. AI-graded and peer-graded options.

Rubric, in order of marks:

1. Data loading + augmentation pipeline, correct train/val/test split
2. Pretrained base loaded with include_top=False, correctly frozen
3. Custom classification head, sensible architecture
4. Stage 1 feature extraction, then Stage 2 fine-tuning at a lower LR
5. Training curves plotted for both stages
6. Evaluation: accuracy, confusion matrix, per-class metrics
7. Sample predictions visualised, including failures
8. Written discussion — what worked, what you'd change

Time-savers:

• EarlyStopping(patience=5, restore_best_weights=True) +

ReduceLROnPlateau(factor=0.2, patience=3) — set both, always

• ModelCheckpoint(save_best_only=True) so a crashed session doesn't cost you the run
• Use the base model's own preprocess_input, not a manual /255, unless the notebook says so
• Discussing your failure cases scores better than a clean 99% with no analysis

---

Course 3 in one page

• Sequential → Functional (branching) → Subclassing (control flow)
• Custom layer: __init__ config, build weights via add_weight, call forward
• Transfer learning: freeze base → train head at 1e-3 → unfreeze top → fine-tune at 1e-5 →

recompile after every trainable change

• Augment train only; pick label-preserving transforms
• Transpose conv upsamples; kernel divisible by stride avoids checkerboarding
• Transformer block = attention → add&norm → FFN → add&norm; LayerNorm not BatchNorm; attention is O(n²)
• AE compresses, VAE generates, GAN is adversarial (mode collapse), diffusion is denoising (stable, slow)
• GradientTape for custom loops; Keras Tuner Hyperband for search
• Quantisation / pruning / distillation for deployment
• RL: Q-learning + experience replay + target network = DQN, linear output