Missing data

Missingness is evidence, not automatically zero

pandas represents missing data with sentinels such as None, NaN or pd.NA, depending on dtype. isna detects missing values; comparing directly with a sentinel is not a reliable test. Empty strings and zero are present values, not missing by default.

isna().sum() counts missing values by column. fillna replaces missing entries with an explicit default; this course permits no note for optional notes, not a fabricated numeric score. Assign the filled notes Series back to its column. dropna(subset=[...]) rejects rows missing only the named required fields. After counting, to_dict converts the resulting Series into an ordinary dictionary of column-name keys and count values. It changes the representation, not the missing-data policy.

An absent required column is a schema problem, not an ordinary missing cell. A row with a missing optional note may still be usable. These demonstrations test missingness only, not whether a present ID or label is valid.

# Missingness is evidence, not automatically zero.
import pandas as pd

df = pd.DataFrame([
    {"item_id": "Museum", "notes": None},
    {"item_id": "Canal", "notes": ""},
])
print(df["notes"].isna().tolist())
df["notes"] = df["notes"].fillna("no note")
print(df["notes"].tolist())

State the policy before filling or dropping data, and retain exclusion counts when building an audit.

Official pandas 2.2 reference for this lesson