Milestone 3 of 12

Debugging, tests, and AI verification

Use AI as a collaborator while independently proving behavior.

Evidence funnel

failuretestverified evidence

Debugging is not a race to produce a plausible edit. It is the disciplined conversion of an observation into a falsifiable hypothesis, a small experiment, and evidence that the repair works without changing neighbouring behaviour.

Required route: diagnostic and visual model (45 min), tracebacks and hypothesis debugging (70 min), logging/breakpoints/assertions (65 min), pytest and contracts (85 min), classes and AI verification (55 min), failure clinic (45 min), transfer/evidence/summary (55 min). Total: 7 hours.

Entry diagnostic

A function that averages frame QC scores crashes on one clip. Before proposing a repair, ask for the exact input, full traceback, runtime and command. Identify the final frame in project code, state the observed exception, and write two competing explanations. Design one minimal input that separates them.

The compressed route requires more than naming ZeroDivisionError: explain which assumption was violated, reproduce it with the smallest case, and name a test that would fail before the repair and pass after it.

Why this matters in CineQC

Media pipelines fail in three different ways. Syntax failures prevent a module from loading. Runtime failures stop on particular clips or metadata. Silent-behaviour failures finish successfully while returning a plausible but wrong QC score. The third is often most expensive because no red error message protects the reviewer.

CineQC-adjacent functions sit at boundaries: paths become decoded clips, clips become frames, frames become scores, and scores become review decisions. Debugging those boundaries requires observations and contracts, especially when an AI assistant authored the proposed change.

Visual model — the evidence funnel

Use one loop for every failure:

1. Observe: command, input, actual output, traceback or log.
2. Specify: expected output and the contract that justifies it.
3. Hypothesize: one possible cause stated so evidence can disprove it.
4. Experiment: smallest changed input or instrumented run that separates causes.
5. Repair: smallest code change supported by the experiment.
6. Verify: targeted regression test, wider suite, diff review, limitations.

Do not change three variables at once. A successful rerun after an unexplained edit is correlation, not diagnosis.

Sandbox lab

Use the isolated local environment from M02 for the required pytest, logging, pdb, and broken-file work. The shared browser Python lab remains a live M01 runner; its editor can serve as an optional scratch runtime for a pure-Python score function, but it has no guided M03 exercise manifest. Copy each broken example into its own local file so an intentional syntax error cannot prevent the other exercises from loading.

Read tracebacks from the failure inward

A traceback records the call path. Read the exception type and message at the bottom, then move upward to the last frame in code you control. Capture filenames, line numbers, function arguments, and the smallest triggering input. Earlier frames explain how execution arrived there; they are not automatically the cause.

Runtime: Local Python 3.12 · save as runtime_failure.py · deliberately raises ZeroDivisionError · 10 min.

def mean_qc_score(scores):
    return sum(scores) / len(scores)

def summarize_clip(clip):
    return {"clip": clip["name"], "mean_score": mean_qc_score(clip["scores"])}

print(summarize_clip({"name": "empty.mov", "scores": []}))

Observation: the final project frame divides by len(scores), which is zero. Expected behaviour is not yet defined. Two hypotheses are possible: empty scores mean “clean,” or empty scores mean “measurement unavailable.” A test cannot choose product semantics for you. For QC evidence, treating missing measurement as clean hides uncertainty, so this milestone makes empty input an explicit contract error.

Self-check

1. Which traceback frame is the first place to inspect?
2. What evidence distinguishes “decoder produced no frames” from “filter removed every score”?
3. Why must expected empty-input behaviour be decided before the fix?

Make observations durable with logging

print is useful for a tiny reproducer. Logging is better when the same context must survive repeated runs. Log identities, counts, shapes, units, thresholds, and branch choices—not private frame content, tokens, or entire production payloads.

Runtime: Local Python 3.12 or browser lab · standard library only · 15 min.

import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger("metadata_qc")

def classify_scores(clip_name, scores, threshold):
    logger.info(
        "classifying clip=%s score_count=%d threshold=%.3f",
        clip_name,
        len(scores),
        threshold,
    )
    if not scores:
        raise ValueError("scores must contain at least one measurement")
    worst = max(scores)
    logger.info("result clip=%s worst=%.3f flagged=%s", clip_name, worst, worst >= threshold)
    return {"worst": worst, "flagged": worst >= threshold}

Use DEBUG for temporary detailed state, INFO for normal lifecycle evidence, WARNING for recoverable degradation, and ERROR when requested work cannot complete. Logging a branch decision often reveals a silent failure faster than printing a final value.

Breakpoints and pdb test a hypothesis interactively

Place breakpoint() just before the suspected transformation, not at the top of the program. Inspect values, step one line, and leave once the hypothesis is answered. Never commit an accidental breakpoint in production code.

Runtime: Local Python 3.12 terminal · pauses in pdb; exit with q without changing files · 10 min.

def neighbor_deltas(frame_scores):
    breakpoint()
    return [abs(current - previous) for previous, current in zip(frame_scores, frame_scores[1:])]

print(neighbor_deltas([0.08, 0.31, 0.12]))

Runtime: Active pdb prompt from the previous example · interactive inspection only · 5 min.

p frame_scores
p frame_scores[1:]
p list(zip(frame_scores, frame_scores[1:]))
n
c

Useful commands are p expression, n for next line, s to step into a call, c to continue, and q to quit. The debugger is an experiment bench; the regression test becomes durable evidence.

Assertions state internal invariants

An assertion catches a programmer assumption close to its source. Public CLI input should receive a clear exception or parser error; it should not depend on assertions, which may be disabled. Use assertions in tests and for internal facts that would indicate a code defect.

Runtime: Local Python 3.12 or browser lab · standard library only · 10 min.

def normalized_qc_score(raw_score, scale):
    if scale <= 0:
        raise ValueError("scale must be positive")
    normalized = raw_score / scale
    assert normalized >= 0, "QC scores cannot be negative after normalization"
    return min(normalized, 1.0)

The ValueError defines a caller-facing contract. The assertion documents an internal invariant after validation. Neither replaces tests across normal, boundary, and failure cases.

pytest turns expectations into executable evidence

pytest discovers functions named test_*, runs each independently, and reports plain-assert values. A useful regression test names the break it catches and uses a hand-derived expected result. Test the observable media contract, not the fact that a helper was called.

Runtime: Local Python 3.12 with pytest · save as test_scores.py, then run python -m pytest -q · 20 min.

import pytest

from scores import classify_scores, mean_qc_score

def test_classify_scores_flags_the_largest_measurement():
    assert classify_scores("shot.mov", [0.08, 0.31, 0.12], 0.25) == {
        "worst": 0.31,
        "flagged": True,
    }

def test_mean_qc_score_rejects_missing_measurements():
    with pytest.raises(ValueError, match="at least one"):
        mean_qc_score([])

@pytest.mark.parametrize(
    "score,threshold,expected",
    [(0.24, 0.25, False), (0.25, 0.25, True), (0.26, 0.25, True)],
)
def test_threshold_boundary_is_inclusive(score, threshold, expected):
    assert classify_scores("shot.mov", [score], threshold)["flagged"] is expected

The three threshold cases protect the boundary rule. pytest.approx is appropriate when a hand-calculated float may differ by harmless rounding; it is not permission to choose an enormous tolerance.

Contract tests protect boundaries

A unit test isolates a small behaviour. A contract test proves the shape and semantics exchanged between components: required JSON keys, score range, path policy, or model input/output shape. The M02 CLI promises a JSON object with stable keys and refuses overwrite.

Runtime: Local Python 3.12 with pytest · uses pytest's tmp_path; creates files only inside the test directory · 20 min.

import json
import pytest

from metadata_qc import write_new_json

def test_report_writer_preserves_the_json_contract(tmp_path):
    output = tmp_path / "report.json"
    payload = {"clip": "shot.mov", "worst_score": 0.31, "flagged": True}

    write_new_json(output, payload)

    assert json.loads(output.read_text(encoding="utf-8")) == payload

def test_report_writer_refuses_to_replace_evidence(tmp_path):
    output = tmp_path / "report.json"
    output.write_text("original", encoding="utf-8")

    with pytest.raises(FileExistsError, match="Refusing to overwrite"):
        write_new_json(output, {"flagged": False})

    assert output.read_text(encoding="utf-8") == "original"

Self-check

1. Which test fails if max(scores) silently becomes min(scores)?
2. Why is the existing-file assertion part of a boundary contract?
3. When should a caller receive ValueError instead of an internal assert failure?

Classes and objects prepare you for PyTorch

A class defines state plus behaviour; an object is one configured instance. Use a class when multiple functions share validated configuration or learned state, not merely to wrap one function. PyTorch models subclass torch.nn.Module: parameters and submodules are registered on the object, forward defines transformation, and calling the object invokes that path. The small pure-Python class below teaches the same state/behaviour contract without requiring a GPU.

Runtime: Local Python 3.12 or browser lab · standard library only · 15 min.

class TemporalThreshold:
    def __init__(self, metric: str, threshold: float):
        if threshold < 0:
            raise ValueError("threshold must be non-negative")
        self.metric = metric
        self.threshold = threshold

    def __call__(self, frame_scores: list[float]) -> dict:
        if not frame_scores:
            raise ValueError("frame_scores must not be empty")
        worst = max(frame_scores)
        return {
            "metric": self.metric,
            "worst": worst,
            "flagged": worst >= self.threshold,
        }

flicker_detector = TemporalThreshold("flicker", 0.25)
print(flicker_detector([0.08, 0.31, 0.12]))

Trace the object: construction stores metric and threshold; the call receives per-frame scores; validation protects the contract; output keeps both measurement and interpretation. Later, a PyTorch module adds tensors, parameters, device state, and training/evaluation modes, but it still needs explicit input and output contracts.

Verify AI-generated changes independently

AI can propose hypotheses, explain tracebacks, draft tests, or review a diff. It cannot observe a run it did not perform, choose unstated product semantics, or turn a passing test into proof of all behaviour. Treat generated code as an untrusted patch.

Use this verification sequence:

1. Preserve the failing command, input, and output before prompting.
2. State the contract and ask the AI to separate observations from assumptions.
3. Inspect the proposed diff for scope, paths, dependencies, secrets, and unrelated edits.
4. Write or review a regression test whose expected value is derived independently.
5. Run the test against the broken version and see the expected failure.
6. Apply the smallest supported repair and rerun the targeted test.
7. Run the wider suite and the original command; review logs and generated artifacts.
8. Explain the important data flow, limitation, and next experiment in your own words.

Reject a patch that changes the contract merely to make its own test pass, catches every exception, invents unseen API fields, disables a check, or claims success without run output.

Self-check

1. What evidence must exist before accepting an AI-authored regression test?
2. Why rerun the original command after the targeted test passes?
3. Which verification step catches an unrelated dependency or secret added to the diff?

Failure clinic

Repair these in order. Keep each broken function in a separate file and record the original failure before editing.

Broken 1 — syntax failure

The missing colon prevents Python from constructing the function. No test inside the file can run until parsing succeeds.

Runtime: Local Python 3.12 · save as broken_syntax.py · deliberately raises SyntaxError: expected ':' · 10 min.

def clip_label(metadata)
    return f"{metadata['clip']}:{metadata['codec']}"

print(clip_label({"clip": "shot.mov", "codec": "prores"}))

Hypothesis: the function header is incomplete. Experiment: use the exact caret in the syntax report, add only the colon, and rerun. Then test the returned label rather than merely testing that the file loads.

Broken 2 — runtime failure

The function parses and works for non-empty input, then divides by zero when decoding or filtering produces no measurements.

Runtime: Local Python 3.12 or browser lab · deliberately raises ZeroDivisionError · 10 min.

def mean_qc_score(scores):
    return sum(scores) / len(scores)

print(mean_qc_score([]))

Hypothesis: empty input violates the measurement contract. Repair by raising a descriptive ValueError, then prove normal input still averages and empty input raises the defined error. Do not silently return 0.0; “not measured” is not “clean.”

Broken 3 — silent-behaviour failure

This function exits successfully and returns a plausible number, but it selects the best score instead of the worst.

Runtime: Local Python 3.12 or browser lab · deliberately prints the wrong value 0.08 · 10 min.

def worst_qc_score(scores):
    if not scores:
        raise ValueError("scores must not be empty")
    return min(scores)

print(worst_qc_score([0.08, 0.31, 0.12]))

Hypothesis: the aggregation direction contradicts the “higher is worse” score contract. Write assert worst_qc_score([0.08, 0.31, 0.12]) == 0.31, observe it fail with 0.08, replace min with max, and run boundary cases. This is why “the script ran” is weak evidence.

CineQC transfer

Assemble the three repaired functions behind one small metadata pipeline. Add structured logs for clip name, score count, threshold, and decision. Add a TemporalThreshold object and contract tests for its returned fields. Change one fixture so it has no scores and prove the pipeline reports a defined failure rather than a clean clip.

Ask an AI assistant for one review of the diff, but require it to list assumptions and suggested tests separately. Accept or reject each suggestion with observed output. Record one useful suggestion and one suggestion you rejected or changed.

Evidence gate

Submit the broken and repaired cases plus a short defence:

1. Reproduce: capture the syntax error, runtime traceback, and silent wrong result before repair.
2. Hypothesize: write one falsifiable cause and smallest distinguishing experiment for each.
3. Repair: show minimal diffs and focused pytest cases that fail on the broken behaviour and pass on the repaired behaviour.
4. Verify: run the targeted tests, wider suite, and original pipeline command; include logs without private media or secrets.
5. Explain: trace metadata to scores to object state to report, state why missing measurement is not clean, and identify a limitation and next experiment.
6. Audit AI: identify which lines were AI-assisted and defend them with independent evidence.

The gate passes only when the artifact works, the tests are relevant, you can diagnose an altered failure, and you can explain what remains unproven. A green suite over synthetic metadata does not prove decoder correctness or real-world detector quality.

One-page summary

• Read the exception and final project frame before editing.
• Turn observations into falsifiable hypotheses and one-variable experiments.
• Log identities, counts, units, thresholds, and branch decisions without leaking private data.
• Use breakpoints to answer a question; use tests to preserve the answer.
• Define caller errors, internal assertions, unit tests, and boundary contracts deliberately.
• Protect normal, boundary, and failure cases; silent wrong values need tests too.
• Classes bind validated state to behaviour and prepare you to reason about PyTorch modules.
• Treat AI output as an untrusted patch: inspect, reproduce, test, rerun, and explain.