Adding a model to a shipped tool without betting the tool on it

Problem

CineQC is a working tool people already rely on. Adding a model to it is easy; adding one that cannot make it worse is the actual engineering problem. A learned signal that overwrites a measurement, or that returns a plausible number when it has in fact failed, damages something that currently works.

Baseline

The baseline is CineQC as it stands: five heuristic scores computed in the browser, plus a human rubric. That behaviour is the floor. The integration is only acceptable if the floor is unchanged when the model is off, unavailable, slow, or unsure.

Data

The browser sends the luma timeline — one value per frame, capped at 32 frames — because that is all the CineQC pipeline retains. It is enough for a temporal signal, and it is why the service keeps semantic confidence low enough to route to human review. Nothing about the payload is logged: frames are customer footage, they may be analysed and never recorded, and a test asserts no pixel value or clip id reaches the log even at debug level.

System

A FastAPI POST /v1/analyze with its bounds in the schema rather than the handler — 2 to 32 frames, at most 256×256, pixels in 0 to 1, uniform geometry, no unknown fields. A bound checked in one code path is a bound the next code path forgets.

• No neutral score on failure. An unavailable model returns 503, a slow one 504, never 0.5. A caller cannot tell a fallback 0.5 from a real measurement, which makes it worse than an error.
• Confidence below 0.55 sets actionable: false and routed_to_review: true in the payload itself, rather than in documentation nobody reads at 2am.
• The service never returns heuristic scores. It receives them as context and does not echo them; a service that restates CineQC's own measurements can silently disagree with them, and then two numbers exist for one thing.
• Timeout and unavailability are distinct. "The model is not here" and "the model did not finish" call for different operator responses.

Evaluation

The property under test is not accuracy, it is containment. Every failure path — feature off, too few frames, no fetch, HTTP error, unknown schema version, timeout — funnels into one shape: { status: 'unavailable', fallback: 'heuristics-human-review', warning }. Learned results attach to state.learned, deliberately not inside state.analysis.scores, and a test asserts the heuristic scores object is unchanged before and after attachment on both the success and the failure path.

An unrecognised schema_version is treated as a failure rather than parsed optimistically: fields may have moved, and a score read from the wrong place is worse than no score at all.

The report carries signalOrigins mapping automated → heuristic, learned → learned, rubric → human. A measurement, a model, and a person are different kinds of claim, and a report that blends them cannot say which kind any given number is.

Failure gallery

The failures worth showing here are the ones engineered out rather than observed:

• A model returning 0.5 on a crash, indistinguishable from a real measurement — replaced by a 503 carrying no score at all.
• A low-confidence score being acted on because its caveat lived in a document — moved into the payload as actionable.
• A response-shape change silently mis-reading a score — refused by exact schema-version match.
• A private-mode browser throwing on localStorage and defaulting the flag to on — any error now reads as off.

Limitations

• The service's two analysers are stand-ins with the real models' signature; the trained checkpoints are not yet served.
• The service has no authentication or rate limiting and is not deployed.
• The browser sends only the luma timeline, which is why semantic confidence stays low enough to route to review.
• Containment is demonstrated by tests, not by production traffic. Nothing here has run against real users.

Next steps

• Serve the trained checkpoints behind the same contract, so model_versions names what actually ran.
• Add authentication and rate limiting before the service is exposed anywhere.
• Send a small downsampled frame rather than the luma timeline, so the semantic signal can rise above the review floor.
• Surface the learned block in the report UI, labelled by origin, once a real model is behind it.

Repository and reproduction

Service and adapter: cineqc-hybrid-api.

Defence

The question I expect is why the flag defaults to off when the whole point was to add a signal. Because the tool is already useful, and the cost of a wrong learned score reaching a delivery decision is far higher than the benefit of an extra advisory number. Off by default is what makes the integration reversible, and reversible is what makes it safe to ship at all.