FrameGuard Sightline

OpenCV 5 checks every frame of an exported video, an agent re-checks each suspicion, and a person approves every fix. Runs on AWS Lambda, Step Functions and S3.

Comment

Inspiration

Exported video still reaches clients with faults that last a fraction of a second: a black frame from a timeline gap, a stalled render, one stray frame from another shot, a frame with block damage, or dropped frames. A tired reviewer misses them, and a second check costs another full viewing. Small studios, animation teams and media labs rarely have a dedicated quality-control operator.

What it does

You upload a rendered or exported clip to a review page. OpenCV 5 measures every frame once. An agent then re-checks each suspicion with the test that fits it and ends each finding as a defect, as dismissed with a reason (a fade, an edit, a held title, slow motion), or as an item for review. Every finding shows the frames around it and the checks behind the verdict.

Nothing changes until a person approves. Approved black, flash and damaged frames are patched in a preview by holding the nearest good frame of the same shot; stalls and skips go to a re-render list, because an export cannot recover lost frames. The agent then analyses its own repair and reports whether every repaired range is clean.

How I built it

  • OpenCV 5.0 (opencv-python-headless 5.0.0.93): VideoCapture, resize, cvtColor, meanStdDev, calcHist and compareHist, a DCT perceptual hash, Canny, an 8x8 block-grid measure, calcOpticalFlowFarneback, absdiff, and VideoWriter for the repair preview.
  • The agent is a bounded controller, not a language model. It owns five checks and a budget of 60 checks and 1,200 re-decoded frames per job; leftovers go to a person. Each check is logged with its trigger, its measurements and its decision.
  • AWS, one CloudFormation template in ap-southeast-2: S3 for uploads and reports (presigned upload, 7-day expiry), a Step Functions standard workflow (analyse, wait for a person with a task token, then act and re-check), arm64 Lambda with OpenCV 5, DynamoDB for job state, an HTTP API throttled to 5 requests per second, and CloudWatch metrics without image content. Every role carries a permissions boundary, and the page accepts at most 10 jobs per UTC day.

Challenges I ran into

  • Farneback optical flow is unstable on flat or saturated regions: on two identical frames of the Sintel trailer, about 1% of vectors reported displacements up to 540 px. Statistics now drop the largest 2% of vectors, and stalls are decided from changed pixels, not from flow.
  • The block-grid ratio explodes on black and plain frames. A damage candidate now also needs a large absolute step, and the follow-up check looks for 16x16 blocks that change in that frame only.
  • The Linux OpenCV wheel has no H.264 encoder, so the repair preview on Lambda is VP8 WebM.

Accomplishments that I'm proud of

  • Thresholds were frozen on development seeds before the test sets were generated. On 24 synthetic test clips (5,349 frames) Sightline found 88 of 93 seeded defects with no false positives. On real trailers with seeded defects (6 clips, 6,184 frames) it found 14 of 18, at 0.97 false positives per 1,000 frames. The unmodified trailers produced no defect findings.
  • The full loop runs on AWS: in the demo job the first pass took 18.3 ms per frame on Lambda (arm64, 2 GB), the agent reported four defects and one item for review, and after approval the re-check of the repaired preview was clean.

What I learned

A detector's raw signal is rarely the decision. In the demo job the follow-up checks dismissed 28 of 33 events, each with its reason, and sent the one case they could not settle to a person instead of guessing.

What's next

Frame skips are the weakest class on real footage (1 of 3 found), and a dark flash is reported as a black frame. Next are skip checks that follow the camera's motion, and measurements on grain, interlaced, HDR and long-take footage.

Links

Credits

Sintel and Big Buck Bunny trailers (c) Blender Foundation, CC BY 3.0, used for evaluation and the demo. The video's narration is a synthetic voice generated locally with Qwen3-TTS.

Built With

Share this project:

Updates

Submission history