Motion graphic

AI Capstone Project with Deep Learning

15 hours · 4 modules · Keras AND PyTorch, side by side Domain: geospatial agricultural land classification from satellite imagery.

No new theory. This course makes you build the same pipeline twice — once in Keras, once in PyTorch — and then compare. The comparison is the assessment. Every module has a "checklist" graded quiz that mirrors the lab rubric, so read the checklist before starting the lab.

---

Module 1 — Data Handling

Memory-based vs generator-based loading

The first lab's entire point.

Generator-based also gives you fresh augmentation every epoch, which is a genuine regularisation benefit, not just a memory workaround.

# memory-based
X = np.array([load(f) for f in files])          # dies at ~10k+ images
# generator-based
gen = datagen.flow_from_directory('data/train', batch_size=32)

Bottleneck to name in your write-up: generator pipelines are usually I/O bound, and the fix is prefetching and parallel workers (num_workers in PyTorch, .prefetch(tf.data.AUTOTUNE) in TensorFlow) so loading overlaps with GPU compute.

Data loading and augmentation — Keras

from tensorflow.keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20, width_shift_range=0.1, height_shift_range=0.1,
    horizontal_flip=True, vertical_flip=True,       # valid for satellite imagery
    zoom_range=0.15,
    validation_split=0.2,
)
train_gen = datagen.flow_from_directory(
    'data', target_size=(224,224), batch_size=32,
    class_mode='categorical', subset='training')
val_gen = datagen.flow_from_directory(
    'data', target_size=(224,224), batch_size=32,
    class_mode='categorical', subset='validation')

Domain point worth making in the write-up: satellite imagery is viewed from directly overhead, so there is no canonical "up". Vertical flips and full 360° rotations are legitimate here, unlike in ordinary photography. Graders reward domain reasoning about augmentation choice.

Data loading and augmentation — PyTorch

train_tf = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomVerticalFlip(),
    transforms.RandomRotation(30),
    transforms.ToTensor(),
    transforms.Normalize([0.485,0.456,0.406], [0.229,0.224,0.225]),
])
val_tf = transforms.Compose([                      # NO augmentation on validation
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize([0.485,0.456,0.406], [0.229,0.224,0.225]),
])

train_ds = datasets.ImageFolder('data/train', transform=train_tf)
val_ds   = datasets.ImageFolder('data/val',   transform=val_tf)
train_dl = DataLoader(train_ds, batch_size=32, shuffle=True,  num_workers=4)
val_dl   = DataLoader(val_ds,   batch_size=32, shuffle=False, num_workers=4)

The two frameworks, side by side

Checklist-quiz points

• Augment train only; validation and test get resize + normalise only
• Class order comes from alphabetical folder names in both frameworks —

train_gen.class_indices / train_ds.class_to_idx. Check it; mismatched class maps between your two models will wreck the comparison.

• Set seeds (np.random.seed, torch.manual_seed, tf.random.set_seed) or the comparison

isn't a comparison.

---

Module 2 — CNN Model Development

Build the same classifier in both frameworks, then compare.

Keras classifier

base = keras.applications.ResNet50(weights='imagenet', include_top=False,
                                   input_shape=(224,224,3))
base.trainable = False

model = keras.Sequential([
    base,
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(num_classes, activation='softmax'),
])
model.compile(optimizer=keras.optimizers.Adam(1e-3),
              loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(train_gen, validation_data=val_gen, epochs=10,
                    callbacks=[keras.callbacks.EarlyStopping(patience=3,
                                                             restore_best_weights=True)])

PyTorch classifier

model = models.resnet50(pretrained=True)
for p in model.parameters():
    p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, num_classes)
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.fc.parameters(), lr=1e-3)

for epoch in range(epochs):
    model.train()
    for x, y in train_dl:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()

    model.eval()
    correct = total = 0
    with torch.no_grad():
        for x, y in val_dl:
            x, y = x.to(device), y.to(device)
            correct += (model(x).argmax(1) == y).sum().item()
            total   += y.size(0)
    print(f'epoch {epoch} val acc {correct/total:.4f}')

Loading and evaluating a pretrained model

The third lab: load a supplied trained model and evaluate it — no training. Tests whether you can reproduce the exact preprocessing the model was trained with. Wrong normalisation gives plausible-looking but degraded accuracy, with no error message. Check the input pipeline first when a loaded model underperforms.

Comparative analysis — what to actually write

This is what's marked, not the accuracy number:

1. Accuracy / F1 per class — a table, both frameworks
2. Training time per epoch and to convergence
3. Memory footprint
4. Convergence behaviour — curve shapes, stability
5. Developer experience — Keras is faster to write; PyTorch gives finer control
6. Where they differ and why — differences almost always trace to preprocessing, default

initialisation, or optimiser defaults, not to the frameworks being differently capable

Honest framing that scores well: with matched preprocessing, matched architecture and matched hyperparameters, the two frameworks should land within noise of each other. A large gap is a bug in your pipeline, not a finding about the frameworks. Say that, then show your matched setup.

Keras default Adam lr is 1e-3; PyTorch default Adam lr is also 1e-3 — but Keras Dense uses Glorot uniform init while PyTorch Linear uses Kaiming uniform. That's a real, citable difference for your write-up.

---

Module 3 — CNN–Vision Transformer Integration

Vision Transformers

Apply the transformer architecture to images by cutting the image into fixed-size patches (typically 16×16), flattening each into a vector, linearly projecting it to an embedding, adding a positional embedding, and feeding the sequence to a standard transformer encoder.

An image becomes a sentence of patches. A 224×224 image with 16×16 patches → 14×14 = 196 tokens, plus a [CLS] token whose final representation is used for classification.

CNN vs ViT

The key trade-off to state in the report: a CNN's inductive bias is a prior. It's a head-start when data is limited and a ceiling when data is abundant. That's why ViTs lose to CNNs on small datasets and beat them at scale.

Why this matters for land classification: CNNs catch local texture (crop rows, field boundaries); ViTs relate distant regions (a river's course, field-layout patterns across the tile). Hybrid models — CNN backbone extracting features, transformer encoder on top — are often the practical best, and that's exactly what this module's final lab builds.

ViT in Keras

Either build from patch-extraction layers (the from-scratch lab) or use a pretrained ViT:

patches = keras.layers.Conv2D(filters=embed_dim, kernel_size=patch_size,
                              strides=patch_size)(inputs)      # patch embedding as a strided conv
patches = keras.layers.Reshape((num_patches, embed_dim))(patches)
x = patches + positional_embedding
for _ in range(num_layers):
    x = TransformerBlock(embed_dim, num_heads, ff_dim)(x)
x = keras.layers.GlobalAveragePooling1D()(x)
out = keras.layers.Dense(num_classes, activation='softmax')(x)

Neat trick worth noting: patch embedding is just a Conv2d with kernel = stride = patch_size. Non-overlapping windows, each linearly projected. Same operation, different framing.

ViT in PyTorch

import timm
model = timm.create_model('vit_base_patch16_224', pretrained=True,
                          num_classes=num_classes)

or via torchvision.models.vit_b_16(weights='DEFAULT'), or Hugging Face ViTForImageClassification.

Note the input size is fixed by the positional embeddings — vit_base_patch16_224 wants exactly 224×224. Resize accordingly, or interpolate the position embeddings.

The integration lab

Combine a CNN feature extractor with a transformer:

image → CNN backbone (e.g. ResNet, minus its head) → feature map (H'×W'×C)
      → flatten spatial dims into a sequence of H'·W' tokens
      → + positional encoding → transformer encoder → pool → classifier

The CNN gives efficient local feature extraction and reduces the sequence length (so attention stays affordable); the transformer adds global reasoning over those features.

Checklist-quiz points

• ViTs need pretraining on this dataset size — training one from scratch on the capstone data

will underperform, and saying so with evidence is the correct answer, not a failure

• Patch count = (H/P) × (W/P); attention cost is O(patches²)
• The [CLS] token is the classification handle; global average pooling over patch tokens is the

common alternative

---

Module 4 — Final Project Submission and Wrap-Up

Deliverable: one notebook/report covering the full pipeline in both frameworks, with the comparative analysis as the centrepiece.

Structure that maps to the rubric:

1. Problem statement — land classification, class list, why it matters
2. Data — source, size, class balance, sample images
3. Pipeline — loading strategy chosen and why; augmentation choices justified for satellite

imagery

1. Keras model — architecture, training, curves, results
2. PyTorch model — same, matched
3. ViT — architecture, results, comparison to the CNNs
4. Comparative analysis — the table: accuracy, per-class F1, epoch time, memory, convergence
5. Error analysis — confusion matrix, actual misclassified tiles shown, which classes

confuse and why in domain terms

1. Conclusion — which model you'd deploy and on what evidence
2. Limitations and next steps

Marks-per-effort ordering

Highest return:

• Error analysis with real images. Most submissions skip it; it's heavily weighted.
• A genuine comparison table with matched conditions.
• Domain reasoning (why vertical flips are valid here; why certain crop classes confuse).

Lowest return:

• Chasing the last 1% of accuracy.
• Elaborate architectures without justification.

Common failure modes

• Different preprocessing between the two frameworks → invalid comparison. Match resize,

normalisation constants and augmentation exactly.

• Class index mismatch → confusion matrices that aren't comparable. Print

class_indices / class_to_idx for both and assert they agree.

• No seed → run-to-run variance mistaken for a framework difference.
• Augmenting the validation set → optimistic and unstable validation scores.
• ViT from scratch on small data, then reporting it as "ViTs are bad". Use pretrained, or

report the result with the correct explanation.

---

Course 6 in one page

• Generator-based loading for anything real; memory-based only for small sets
• Satellite imagery: vertical flips and full rotations are legitimate — say why
• Same architecture + same preprocessing → the two frameworks should match; a gap means a bug
• Keras HWC vs PyTorch CHW; rescale=1./255 vs ToTensor(); flow_from_directory vs ImageFolder
• ViT = image as a sequence of patches; patch embedding is a strided conv; [CLS] token classifies
• CNN inductive bias helps on small data, caps you on large data
• Hybrid CNN→transformer gets local features cheaply and global reasoning on top
• The comparative analysis and the error analysis carry the marks, not the accuracy number