Milestone 10 of 12

One semantic detector

Choose a defensible semantic target after the temporal benchmark.

Semantic selection

fixed matrixone evaluatorbounded claim

Choose one semantic signal with a rule written before experimentation. Required route: selection audit (75 min), evaluator trade-offs (60 min), fixed-set build (95 min), baseline and errors (80 min), transfer/evidence (75 min), selected reading (35 min). Total: 7 hours.

Entry diagnostic

Given three candidate categories, apply every criterion before discussing a model. State which target qualifies and the precise fallback when none does.

Self-check

Why is a category with a high average but one criterion scored 2/5 ineligible?

Why this matters in CineQC

CineQC needs a useful, lawful signal—not a fashionable demo. The target must have repeatable labels, enough examples, a feasible baseline, and a claim that survives inspection. Selection evidence prevents quietly choosing the category that produced the most flattering early result.

Visual model — fixed selection matrix

Use exactly three candidates: text integrity, anatomy integrity, and cross-frame object continuity. Score each 1–5 on exactly five criteria: available lawful examples, label consistency, domain value, baseline feasibility, and ability to produce at least 200 labeled examples.

Choose the highest-scoring category that achieves at least 3/5 on all five criteria. Resolve a tied total with the higher domain-value score, then baseline-feasibility score, and record that deterministic tie-break without changing eligibility. If no category qualifies, use text integrity with controlled generated samples and clearly limit the claim to visible on-screen text corruption. Do not expand that fallback into language understanding, factuality, typography quality, or all text failures.

Runtime: Local Python 3.12 · standard library · CPU-safe · 15 min.

CRITERIA = ("lawful", "labels", "value", "baseline", "two_hundred")

def select_target(scores):
    eligible = [
        name for name, row in scores.items()
        if all(row.get(criterion, 0) >= 3 for criterion in CRITERIA)
    ]
    if not eligible:
        return {"selected": "text integrity",
                "claim": "visible on-screen text corruption only",
                "fallback": True}
    selected = max(eligible, key=lambda name: (
        sum(scores[name][criterion] for criterion in CRITERIA),
        scores[name]["value"], scores[name]["baseline"], name
    ))
    return {"selected": selected, "claim": "selected lawful category",
            "fallback": False}

scores = {
    "text integrity": dict(zip(CRITERIA, [5, 4, 4, 5, 5])),
    "anatomy integrity": dict(zip(CRITERIA, [3, 2, 5, 2, 3])),
    "cross-frame object continuity": dict(zip(CRITERIA, [4, 3, 5, 3, 4])),
}
print(select_target(scores))

Commit the completed selection matrix, example-count audit, exclusions, tie-break record, and selection date before building the evaluator.

Self-check

What two records prove that “200 examples” means labeled lawful examples rather than scraped files of unknown provenance?

Evaluator routes, not a shopping list

A compact classifier is fast and measurable but needs representative labels. An embedding route can compare examples with a small head but inherits the embedding's blind spots. An OCR-assisted route exposes text boxes and recognized strings, making text corruption inspectable, while OCR errors can masquerade as corruption. A VLM evaluator can express richer prompts but raises latency, cost, privacy, nondeterminism, and validation burdens.

Choose one route; the other three remain documented alternatives. Compare candidates on fixed-test performance, baseline lift, latency distribution, hardware, privacy path, per-item cost, inspectability, and failure behavior. A simple OCR confidence or embedding-distance baseline is enough when matched to the selected target.

Sandbox lab

Freeze at least 200 labeled examples into source-grouped train/validation/test partitions and preserve one fixed test set. Build the smallest evaluator that answers the selected claim. Save per-item IDs, scores, labels, model/config identity, latency, and baseline output. Produce a selection matrix, confusion counts, representative false positives and false negatives, and an error analysis organized by production consequence.

For the fallback, generate controlled clean/corrupted on-screen text pairs from fonts and layouts you may lawfully use. Keep source templates and corruption recipes. The limitation statement must say visible on-screen text corruption only and identify failures such as stylized lettering, low contrast, motion blur, and unseen scripts.

Failure clinic

Repair these invalid routes: averaging away a 2/5 criterion; changing criteria after results; counting unlabeled or unlawfully sourced examples; tuning on the fixed test set; comparing systems on different items; requiring all four evaluator families; sending private frames to a hosted VLM without approval; and calling OCR disagreement “semantic truth.” Rebuild the matrix or narrow the claim rather than hiding the failure.

CineQC transfer

Export an evaluator contract with selected category, claim, input shape/media rules, output score meaning, uncertainty route, model and dataset versions, baseline identity, fixed-test checksum, latency envelope, and privacy mode. It is one additional signal for CineQC; it does not supersede the temporal detector or the human rubric.

Self-check

Which artifact makes a later model change comparable: the fixed test-set IDs plus saved per-item predictions, or a screenshot of one successful example?

Evidence gate

The semantic evaluator passes when a reviewer can inspect the five-column selection matrix, lawful-example audit, at least 200 labeled examples (or the exact fallback), fixed test set, named baseline, saved predictions, measured latency, error analysis, and limitation statement. The reviewer must be able to reproduce the selected rule and see why alternatives were not required.

One-page summary

• Candidates are text integrity, anatomy integrity, and cross-frame object continuity.
• Score lawful examples, label consistency, domain value, baseline feasibility, and 200-example ability.
• Select the highest total only among categories scoring at least 3/5 everywhere.
• If none qualifies, use controlled text samples and claim visible on-screen text corruption only.
• Implement one evaluator route; compare the others on trade-offs.
• Freeze the test set, preserve predictions, compare a baseline, and show errors.