Milestone 7 of 12

Dataset, labels, splits, and leakage

Design a benchmark that earns trust.

Leakage-safe benchmark

lawful sourcegrouped splitdataset card

Build the benchmark before the model. Required route: task and ontology (70 min), agreement and grouping (70 min), split clinic (80 min), dataset build (100 min), failure clinic (40 min), transfer/evidence (60 min). Total: 7 hours, including the selected reading.

Entry diagnostic

A hundred clips become ten thousand frames. A random frame split scores 98%. Identify the likely leakage route and name the grouping key that repairs it.

Self-check

Which question must be answered before choosing labels: clip classification, frame localization, or a temporal score?

Why this matters in CineQC

A detector inherits its dataset's boundaries. CineQC needs labels tied to visible evidence, source-aware splits, and provenance that permits lawful use. A high score cannot repair private-media exposure, duplicate leakage, or an ambiguous ontology.

Visual model — source to frozen benchmark

A split diagram contrasts clip-a leakage across train and test with source-grouped partitions.

Use lawful source → sequence group → anchored label → frozen split → dataset card. Define one primary task: classify a short sequence as clean, flicker, stutter, color_drift, sharpness_collapse, or motion_incoherence. Frame localization and continuous temporal scoring are useful extensions, but are optional here.

A label ontology names each class, inclusion/exclusion rules, boundary cases, and anchored examples. An anchor pairs a short lawful sequence description with the reason for its label. Keep intentional cuts, flashes, holds, focus pulls, and camera moves as explicit clean or review examples so the model is not taught that cinema is a defect.

Measure agreement on a double-labelled subset. Report raw agreement and per-class disagreements; Cohen's kappa may supplement, never replace, the disagreement review. Resolve the guide—not merely the labels—when annotators interpret the same case differently.

Group first, then split

Assign every item a stable source_id and sequence_id. All frames, crops, encodes, and augmented variants derived from one sequence stay in one split. Source grouping is stricter when one master yields several edits. The roles are distinct:

• train fits weights and may receive train-only augmentation;
• validation selects configuration and threshold candidates;
• test is the fixed test set, opened once for final comparison and never used to tune;
• a private evaluation may remain private, but its aggregate method and public/private boundary must be documented.

Deduplicate before splitting using provenance plus content fingerprints. Handle imbalance with documented sampling or loss weights measured on training data. Apply spatial or photometric augmentation only when it preserves the label; never create validation/test variants or let a synthetic defect cross splits.

Runtime: Local Python 3.12 · standard library · 20 min.

from collections import defaultdict

records = [
    {"item": "a-001", "source_id": "master-a", "sequence_id": "clip-a", "label": "clean"},
    {"item": "a-002", "source_id": "master-a", "sequence_id": "clip-a", "label": "flicker"},
    {"item": "b-001", "source_id": "master-b", "sequence_id": "clip-b", "label": "clean"},
]
split_for_source = {"master-a": "train", "master-b": "test"}
groups = defaultdict(set)
for row in records:
    groups[row["source_id"]].add(split_for_source[row["source_id"]])
assert all(len(splits) == 1 for splits in groups.values())
print({row["item"]: split_for_source[row["source_id"]] for row in records})

Self-check

Why is augmenting first and randomly splitting later unsafe even when filenames differ?

Sandbox lab

Create dataset-v1.jsonl from a small public or generated fixture. Validate required fields: item ID, source and sequence groups, label, time range, provenance, license, consent/privacy note, checksum, and split. Freeze the manifest and its checksum before training. Produce a count table by label and split and an automated assertion that no source, sequence, or exact duplicate crosses splits.

Failure clinic — same clip split

The failure: adjacent frames 100–149 from one clip land in train while frames 150–199 land in test. Compression texture, camera, set, and neighboring motion make them near-duplicates. The model recognizes the clip rather than the artifact.

Repair it by grouping every derived frame and window by the parent sequence before the split. If multiple sequences share one master, performer, template, or generation prompt, decide whether the intended generalization requires grouping at that source level too. Re-run duplicate checks after every ingest.

Other failures include changing the ontology after inspecting test errors, treating missing provenance as public permission, allowing aggressive augmentation to erase the defect, and hiding minority-class failures behind accuracy. Preserve lawful licensing and privacy constraints; do not place private CineQC media, paths, thumbnails, or metadata in the public artifact.

CineQC transfer

Map each learned label to the closest deterministic signal and list non-equivalences. A heuristic threshold is not a ground-truth label. Export only sanitized IDs and aggregate counts from private evaluation. Keep a public fixture sufficiently small to reproduce the dataset checks without revealing proprietary media.

Self-check

What makes dataset v1 genuinely frozen: a filename, or immutable records plus hashes, split assignments, label-guide version, and provenance?

Evidence gate

Submit frozen dataset v1, its label guide, and dataset card. Evidence must include:

1. a primary task and six-class ontology with anchored positive, negative, and boundary examples;
2. an agreement sample and documented resolutions;
3. source/sequence-aware train, validation, and fixed test splits with duplicate checks;
4. train-only augmentation and imbalance policy;
5. manifest checksum, provenance, lawful licensing, and privacy decisions;
6. a public fixture and explicit public/private boundary—never private CineQC media.

One-page summary

• Formulate one primary task before labelling.
• Anchor labels in observable sequence evidence.
• Group sources and sequences before splitting.
• Freeze the test set before model decisions.
• Keep augmentation in training and label-preserving.
• Document provenance, licensing, privacy, and limitations.
• A dataset card makes decisions inspectable; it does not make data lawful by itself.