Evaluation metrics field guide
Counts, curves, calibration, and reproducible benchmark evidence.
Use this card while building the benchmark; it is a lookup, not another course.
Counts first
- TP: defect correctly flagged.
- FP: clean/intentional content incorrectly flagged.
- FN: defect missed.
- TN: clean content passed.
precision = TP/(TP+FP), recall = TP/(TP+FN), F1 = 2PR/(P+R). Define zero-denominator behavior in code. Name macro, micro, or per-class averaging.
Curves and calibration
A PR curve recomputes precision and recall over thresholds. Choose a threshold on validation data using documented production cost; report once on the fixed test set. Calibration compares score bins with observed frequencies. Call an uncalibrated output a score, not a probability.
Evidence checklist
Keep saved item-level predictions, dataset/system identities, threshold policy, executable metric code, a confusion matrix, PR points, error galleries, and limitations. The report must be regenerated from predictions rather than hand-entered numbers.
Your local study notes mount here when JavaScript is available.
Teaching modes mount here when JavaScript is available. The readable course remains available without them.