Motion graphic

Machine Learning with Python

20 hours · 6 modules · scikit-learn throughout Final project: Rainfall Prediction Classifier. Practice project: Titanic survival.

This is the foundation course. Everything is scikit-learn. No neural networks yet.

---

Module 1 — Introduction to Machine Learning

Core ideas

What ML is: a program that improves its output on a task by being shown examples, rather than by being given explicit rules. You supply data + a loss function; the algorithm searches for parameters that minimise the loss.

The three families:

Supervised splits again:

• Regression — predict a number (house price, rainfall in mm)
• Classification — predict a category (rain / no rain)

Machine Learning Model Lifecycle

The course teaches this as a loop, and it shows up on the quiz:

1. Problem definition — what decision does this model change?
2. Data collection — sourcing, licensing, coverage
3. Data preparation — cleaning, encoding, scaling, splitting (this is 70–80% of real work)
4. Model development / selection — pick algorithm, train
5. Model evaluation — metrics on held-out data
6. Model deployment — serving, API, batch
7. Model monitoring — drift, retraining

Key exam point: the lifecycle is iterative, not linear. Monitoring feeds back to step 1.

Data Scientist vs AI Engineer

The distinction IBM cares about (they quiz it):

The whole certificate is an argument that the second column is the growth role.

Tools for Machine Learning

• Data: pandas, NumPy
• Visualisation: matplotlib, seaborn
• Classical ML: scikit-learn
• Deep learning: TensorFlow/Keras, PyTorch
• NLP: NLTK, spaCy, Hugging Face
• Big data: Apache Spark / PySpark

The scikit-learn ecosystem

Every scikit-learn model obeys the same interface. Learn this once and you know all of them:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler  = StandardScaler().fit(X_train)   # fit on TRAIN only
X_train = scaler.transform(X_train)
X_test  = scaler.transform(X_test)        # transform, never re-fit

model = LogisticRegression()
model.fit(X_train, y_train)               # .fit()     — learn
y_pred = model.predict(X_test)            # .predict() — infer
model.score(X_test, y_test)               # .score()   — quick metric

fit / transform / fit_transform / predict / score — that's the entire API surface.

---

Module 2 — Linear and Logistic Regression

Linear regression

Simple (one feature): ŷ = θ₀ + θ₁x Multiple (n features): ŷ = θ₀ + θ₁x₁ + … + θₙxₙ = θᵀx

Trained by minimising Mean Squared Error:

MSE = (1/m) · Σ (yᵢ − ŷᵢ)²

Plain English: average of the squared vertical distances from each point to the line. Squared, so that positive and negative errors don't cancel, and so big misses hurt disproportionately.

Two ways to solve it:

• Ordinary Least Squares / normal equation — closed form, θ = (XᵀX)⁻¹Xᵀy. Exact, but the

matrix inversion is O(n³) in the number of features, so it dies on wide data.

• Gradient descent — iterative, scales. Course 2 onward uses this exclusively.

Polynomial and non-linear regression

You keep the linear machinery and change the features:

from sklearn.preprocessing import PolynomialFeatures
X_poly = PolynomialFeatures(degree=3).fit_transform(X)   # x → [1, x, x², x³]

The model is still linear in the parameters — that's why it still works. This is the key conceptual point: "linear regression" means linear in θ, not a straight line in x.

Other non-linear forms the course mentions: exponential, logarithmic, sinusoidal — same trick, transform the feature.

Logistic regression

Used for classification, despite the name. You want a probability in [0, 1], so you squash the linear output through the sigmoid (logistic) function:

σ(z) = 1 / (1 + e^(−z))        where z = θᵀx
P(y=1 | x) = σ(θᵀx)

Sigmoid maps (−∞, +∞) → (0, 1). z = 0 gives 0.5. Decision threshold defaults to 0.5 but is tunable — moving it trades precision against recall.

Why not MSE for classification? MSE + sigmoid gives a non-convex loss surface full of local minima. Instead use log loss / binary cross-entropy:

J(θ) = −(1/m) · Σ [ yᵢ·log(ŷᵢ) + (1−yᵢ)·log(1−ŷᵢ) ]

Read it as: if the true label is 1, you pay −log(ŷ) — which is 0 when ŷ=1 and explodes toward ∞ as ŷ→0. It punishes confident wrong answers brutally. That's the whole idea.

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(C=0.01, solver='liblinear')
clf.fit(X_train, y_train)
yhat       = clf.predict(X_test)         # hard labels
yhat_proba = clf.predict_proba(X_test)   # [P(class0), P(class1)] per row

C is inverse regularisation strength. Small C = strong regularisation. This trips people up.

---

Module 3 — Building Supervised Learning Models

Decision Trees

A tree of if/else splits. At each node the algorithm asks: which feature, at which threshold, best separates the classes? "Best" is measured by impurity reduction.

Two impurity measures:

Gini      = 1 − Σ pᵢ²
Entropy   = − Σ pᵢ · log₂(pᵢ)
Information Gain = entropy(parent) − weighted avg entropy(children)

Both are 0 when a node is pure (all one class), maximal when classes are evenly mixed. Gini is the sklearn default and is slightly cheaper; entropy is what the videos derive.

from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(criterion='entropy', max_depth=4)

Regression trees — same structure, but each leaf predicts the mean of the samples in it, and splits are chosen to minimise variance / MSE rather than Gini.

The course has an Errata video on the Regression Trees lesson. Watch it — the original video has an error, and the corrected version is what the quiz follows.

Trees: interpretable, need no scaling, handle mixed feature types. But high variance — a slightly different training set gives a very different tree. That's what ensembles fix.

Support Vector Machines (SVM)

Find the hyperplane that separates classes with the maximum margin — the widest possible gap. Points sitting on the margin are the support vectors; only they determine the boundary.

For non-linearly-separable data, use the kernel trick: implicitly map into a higher dimension where a linear separator exists, without ever computing the coordinates.

Kernels: linear, poly, rbf (Gaussian, the default), sigmoid.

from sklearn.svm import SVC
svm = SVC(kernel='rbf', C=1.0, gamma='scale')

• C — penalty for misclassification. High C = narrow margin, tries hard to classify every

training point (risk of overfit). Low C = wider, softer margin.

• gamma — how far a single training point's influence reaches. High gamma = tight, wiggly

boundary.

SVMs must have scaled features. They're strong on high-dimensional data (text) but scale badly to very large row counts.

K-Nearest Neighbours (KNN)

No training at all — "lazy learning". To classify a new point, find its k closest training points and take a majority vote (or mean, for regression).

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)

• Distance is usually Euclidean → scaling is mandatory, or the largest-range feature

dominates entirely.

• Small k = low bias, high variance (noisy boundary). Large k = smooth, may underfit.
• Choose k by validation curve. Odd k for binary problems avoids ties.
• Suffers from the curse of dimensionality — in high dimensions everything is roughly

equidistant and "nearest" stops meaning anything.

Bias, Variance and Ensembles

Bias = error from wrong assumptions; the model is too simple (underfitting). Variance = error from sensitivity to the training sample; too complex (overfitting).

Total error = Bias² + Variance + Irreducible noise

You cannot drive both to zero; you trade them. Ensembles are the trick that buys you a better trade than a single model.

Bagging (Bootstrap AGGregatING) — train many models on random resamples with replacement, average their predictions. Reduces variance. Models train independently → parallelisable.

Random Forest = bagging + an extra twist: at each split, only a random subset of features is considered. This decorrelates the trees, which is what makes the averaging actually work.

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, max_features='sqrt', random_state=42)

Boosting — train models sequentially, each one focusing on what the previous got wrong. Reduces bias. Cannot be parallelised across rounds.

XGBoost — gradient boosting done efficiently; the workhorse for tabular data.

from xgboost import XGBClassifier
xgb = XGBClassifier(n_estimators=100, learning_rate=0.1, max_depth=3)

Memorise this pairing — it's a guaranteed quiz question:

Bagging → reduces variance, parallel. Boosting → reduces bias, sequential.

---

Module 4 — Building Unsupervised Learning Models

K-Means

Partition data into k clusters by minimising within-cluster sum of squares (inertia).

The algorithm (Lloyd's):

1. Pick k initial centroids
2. Assign each point to the nearest centroid
3. Move each centroid to the mean of its assigned points
4. Repeat 2–3 until assignments stop changing

from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
labels = km.fit_predict(X)

• k-means++ seeds centroids far apart — much better than random init, and it's the default.
• n_init runs the whole thing multiple times and keeps the best; K-Means converges to a

local minimum, not the global one.

• Assumes clusters are spherical and similar in size. Elongated or nested clusters break it.
• Requires scaling. Requires you to choose k in advance.

Choosing k — the elbow method: plot inertia vs k, look for the bend. Or silhouette score (Module 5).

DBSCAN and HDBSCAN

DBSCAN — density-based. Doesn't need k. Two parameters:

• eps — neighbourhood radius
• min_samples — points required within eps to be a "core point"

Points are classified as core / border / noise. Noise is a real output — DBSCAN is the one that can say "this point belongs to nothing", which K-Means never can.

Finds arbitrarily-shaped clusters. Struggles when cluster densities vary a lot, because one global eps can't fit both.

HDBSCAN — hierarchical DBSCAN. Runs DBSCAN across all eps values and extracts the most stable clusters. Removes the hardest hyperparameter. Handles varying density. This is the one you should reach for in practice.

from sklearn.cluster import DBSCAN, HDBSCAN
db = DBSCAN(eps=0.5, min_samples=5)
hdb = HDBSCAN(min_cluster_size=5)

Dimension reduction

PCA — find the orthogonal directions of maximum variance, project onto the top few. Linear. Deterministic. Preserves global structure. Components are ranked by explained variance.

from sklearn.decomposition import PCA
pca = PCA(n_components=0.95)     # keep enough components for 95% of variance
X_reduced = pca.fit_transform(X_scaled)
pca.explained_variance_ratio_

Must scale before PCA — otherwise the feature with the biggest units dominates every component.

t-SNE — non-linear, preserves local neighbourhoods. Excellent for 2D visualisation, and that's all it's for: distances between clusters in a t-SNE plot are not meaningful, and it's stochastic (different runs, different pictures). Never feed t-SNE output into a classifier.

UMAP — non-linear, faster than t-SNE, preserves more global structure, and unlike t-SNE it can transform() new data. Increasingly the default.

---

Module 5 — Evaluating and Validating Machine Learning Models

This is the most examinable module in the course. Know the confusion matrix cold.

Classification metrics

                Predicted
                 Neg   Pos
Actual   Neg  |  TN  |  FP  |   ← FP = false alarm      (Type I error)
         Pos  |  FN  |  TP  |   ← FN = missed detection (Type II error)

Accuracy  = (TP + TN) / total          "how often am I right"
Precision = TP / (TP + FP)             "when I say yes, how often am I right"
Recall    = TP / (TP + FN)             "of all real yeses, how many did I catch"
F1        = 2 · (P · R) / (P + R)      harmonic mean — punishes imbalance between P and R
Specificity = TN / (TN + FP)           "of all real noes, how many did I catch"

When accuracy lies: 99% of transactions are legitimate → a model that always predicts "legitimate" scores 99% accuracy and catches zero fraud. On imbalanced data use precision, recall, F1, or AUC.

Precision vs recall is a business decision, not a technical one:

• Cancer screening → maximise recall (a missed case is fatal; a false alarm is a second test)
• Spam filter → maximise precision (a real email in the spam folder is worse than spam in

the inbox)

ROC curve — TPR (recall) vs FPR, swept across every decision threshold. AUC is the area under it: 1.0 perfect, 0.5 random. AUC is threshold-independent, which is why it's the standard single-number comparison.

For heavily imbalanced data, the precision-recall curve is more informative than ROC.

Regression metrics

MAE  = (1/m) Σ |y − ŷ|            same units as y, robust to outliers
MSE  = (1/m) Σ (y − ŷ)²           punishes big errors, units are y²
RMSE = √MSE                       same units as y, still outlier-sensitive
R²   = 1 − SS_res/SS_tot          fraction of variance explained; 1.0 perfect, can go negative

R² always increases when you add features, even useless ones — that's why adjusted R² exists (it penalises feature count).

Evaluating clustering (no labels)

• Silhouette score — for each point, (b − a)/max(a, b) where a = mean distance to its own

cluster, b = mean distance to the nearest other cluster. Range [−1, 1]; near 1 is well-clustered, near 0 is on a boundary, negative means it's probably in the wrong cluster.

• Davies-Bouldin — lower is better.
• Elbow / inertia — for K-Means specifically.

Cross-validation

A single train/test split is one noisy estimate. K-fold CV splits the data k ways, trains k times, and averages — a much more stable estimate of generalisation.

from sklearn.model_selection import cross_val_score, StratifiedKFold
scores = cross_val_score(model, X, y, cv=StratifiedKFold(5), scoring='f1')
scores.mean(), scores.std()

• Stratified k-fold keeps class proportions in each fold — always use it for classification.
• Leave-One-Out — k = n. Nearly unbiased, wildly expensive, high variance.
• TimeSeriesSplit — for temporal data. Never shuffle time series; you'd be training on the

future to predict the past.

Regularisation

Penalise large coefficients to stop the model contorting itself around the training data.

Ridge (L2):  Loss + α·Σθⱼ²      shrinks coefficients toward 0, never exactly 0
Lasso (L1):  Loss + α·Σ|θⱼ|     drives some coefficients to exactly 0 → feature selection
ElasticNet:  both, mixed by l1_ratio

α (alpha) is the strength. α = 0 → plain regression. Large α → everything shrinks toward zero and you underfit. In LogisticRegression and SVC the equivalent knob is C = 1/α — inverted.

Scale your features before regularising. The penalty is applied per-coefficient, so unscaled features get penalised unequally and arbitrarily.

Data leakage and other pitfalls

Leakage = information from outside the training set sneaking into training. Symptom: suspiciously excellent validation scores that collapse in production.

Common sources:

1. Scaling/imputing on the full dataset before splitting ← the most common
2. A feature that encodes the target (e.g. discharge_date when predicting admission outcome)
3. Duplicate rows landing in both train and test
4. Time-travel: using future information in a time series
5. Tuning hyperparameters on the test set, then reporting the test score

The fix is Pipeline. A Pipeline makes every preprocessing step fit only inside the training fold of each CV split. This is the single most important practical lesson in the course:

from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=10)),
    ('clf',    RandomForestClassifier(random_state=42)),
])

grid = GridSearchCV(
    pipe,
    param_grid={'clf__n_estimators': [50, 100, 200],
                'clf__max_depth':    [None, 5, 10]},
    cv=5, scoring='f1', n_jobs=-1,
)
grid.fit(X_train, y_train)
grid.best_params_, grid.best_score_

Note the stepname__paramname double-underscore syntax — it's how GridSearchCV addresses a parameter inside a pipeline step. Labs expect it.

Other pitfalls the module names: imbalanced classes, overfitting to the validation set through too many tuning rounds, non-representative sampling, and reporting a single metric.

---

Module 6 — Final Project and Exam

Practice project: Titanic Survival Prediction — classic tabular classification. Mixed categorical/numeric features, missing values in Age and Cabin. Build a ColumnTransformer + Pipeline; that's the whole point of the exercise.

Final project: Rainfall Prediction Classifier — Australian weather data, binary "RainTomorrow" target, imbalanced classes.

Marking rubric essentials:

1. EDA with actual commentary, not just plots
2. Correct train/test split before any preprocessing
3. Pipeline with ColumnTransformer (numeric → impute + scale, categorical → impute + one-hot)
4. At least two model families compared
5. GridSearchCV or RandomizedSearchCV for tuning
6. Metrics appropriate to imbalance — not accuracy alone
7. Confusion matrix, and a paragraph interpreting it in domain terms
8. Feature importance / coefficients discussed

Skeleton that satisfies most of the rubric:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

num_cols = X.select_dtypes(include='number').columns
cat_cols = X.select_dtypes(exclude='number').columns

pre = ColumnTransformer([
    ('num', Pipeline([('imp', SimpleImputer(strategy='median')),
                      ('sc',  StandardScaler())]), num_cols),
    ('cat', Pipeline([('imp', SimpleImputer(strategy='most_frequent')),
                      ('oh',  OneHotEncoder(handle_unknown='ignore'))]), cat_cols),
])

pipe = Pipeline([('pre', pre), ('clf', RandomForestClassifier(random_state=42))])

handle_unknown='ignore' matters — a category present only in test would otherwise crash predict.

Final exam focus areas

• Confusion matrix arithmetic (they give you numbers, you compute precision/recall/F1)
• Which algorithm for which data shape
• Bias/variance diagnosis from train-vs-test scores
• Scaling: which algorithms require it (SVM, KNN, logistic, PCA, K-Means) and which don't

(trees, forests, boosting)

• Bagging vs boosting
• Identifying leakage in a described workflow

---

Course 1 in one page

• Every sklearn model: fit → predict → score
• Split first, preprocess inside a Pipeline, tune with GridSearchCV
• Linear regression = MSE; logistic regression = sigmoid + log loss
• Trees split on impurity; forests = bagging (↓variance); XGBoost = boosting (↓bias)
• Scale for: SVM, KNN, logistic, PCA, K-Means. Don't need to for: trees, RF, XGBoost
• K-Means needs k and assumes spheres; DBSCAN/HDBSCAN don't and find noise
• PCA for preprocessing, t-SNE/UMAP for looking at data
• Accuracy lies on imbalanced data — use precision/recall/F1/AUC
• Leakage is the failure mode that looks like success