# Temporal Detector — benchmark capstone

A small learned defect detector, trained on the shared synthetic fixtures and
benchmarked head-to-head against the Artifact Lab heuristics on the same
held-out clips.

**The heuristics win.** See [MODEL_CARD.md](MODEL_CARD.md) for the numbers and
why that is the interesting result rather than a disappointing one.

- [DATASET_CARD.md](DATASET_CARD.md) — what the data is and what it is not
- [MODEL_CARD.md](MODEL_CARD.md) — architecture, the recorded GPU run, limits
- `runs/local-rtx4080/` — the committed evidence for the published metrics

## Run it

```bash
uv run --project creative-ai-engineering/capstones/temporal-detector python -m temporal_detector.train --out runs/local-rtx4080 --device cuda
uv run --project creative-ai-engineering/capstones/temporal-detector python -m temporal_detector.benchmark --run runs/local-rtx4080
uv run --project creative-ai-engineering/capstones/temporal-detector pytest creative-ai-engineering/capstones/temporal-detector -q
```

`--device cuda` exits non-zero if CUDA is unavailable instead of silently falling
back, so a CPU run cannot be filed as GPU evidence. `--device cpu` runs the same
path without a GPU.

Score one clip with a trained checkpoint:

```bash
uv run --project creative-ai-engineering/capstones/temporal-detector python -m temporal_detector.infer scene-a-flicker --checkpoint runs/local-rtx4080/checkpoint.pt --threshold 0.4975
```

## Layout

```
configs/baseline.yaml     seed, epochs, window, split sizes — no test-set knob
src/temporal_detector/
  data.py                 fixtures -> grouped windows; the split rule lives here
  model.py                the network, checkpoint save/load, hashing
  train.py                training + the run record that makes it evidence
  evaluate.py             predictions -> metrics, confusion matrix, error gallery
  benchmark.py            learned vs heuristic on the same held-out clips
  infer.py                score a single clip
notebooks/cloud_baseline.py    source of truth for the notebook
notebooks/cloud-baseline.ipynb generated; regenerate with scripts/sync_notebook.py
runs/local-rtx4080/            committed evidence
```

## Decisions worth knowing

**The split groups by scene, never by clip or window.** Windows overlap in time
and all six variants of a scene share its background. Split below the scene and a
model scores well by recognising content it already trained on. Two tests pin
this, one of them specifically for windows of a single clip.

**The config cannot reach the test set.** No test threshold, no "maximise this on
test", no retry count keyed to test performance. A test asserts that no config
key mentions the test split, because anything that *can* reach the held-out set
eventually will, and every published number becomes optimistic.

**The threshold comes from validation only.** `choose_threshold` raises if handed
test predictions. Hyperparameters were chosen by a validation-only sweep
(learning rate and epochs; validation F1 plateaued at 0.926 across every setting
tried, so the cheapest of the tied best was taken).

**The benchmark recalibrates the heuristics without the held-out scenes.** The
published Artifact Lab thresholds are fitted on all 48 clips, which include the
12 held out here. Comparing against those would hand the heuristics a look at the
test set that the model never got. It also imports and runs the heuristics rather
than quoting their published numbers — a benchmark that copies the other side's
results cannot notice when they stop matching.

**Metrics are never recomputed by hand.** Both sides go through
`shared.metrics.recompute_classification_metrics` over saved `PredictionRecord`
files, so the number in this repo and the number a reader derives from the
committed predictions are the same computation.

## Tests

```bash
uv run --project creative-ai-engineering/capstones/temporal-detector pytest creative-ai-engineering/capstones/temporal-detector -q
```

39 tests: split grouping and leakage, seed reproducibility, window tensor shape
and range, config hygiene, CPU forward pass, checkpoint save/load prediction
parity, checkpoint hashing, metric recomputation from saved predictions,
validation-only thresholds, confusion matrix and error gallery contents, and the
benchmark's score-normalisation invariants.