Eyeline — Autonomous On-Set Continuity Copilot

Autonomous multimodal continuity copilot for film script supervisors: catches prop, wardrobe, and blocking breaks in the 2-minute on-set window, backed by Gemini 3.8 Flash adjudication and Veo 3.1.

Comment

1. Inspiration: The 2-Minute On-Set Window

On any professional film production, a script supervisor sits beside the director with a physical binder, Polaroid/iPad reference photos, and stopwatches, tracking hundreds of micro-details across coverage:

  • Did the actor pick up the glass with their left hand or right?
  • How much liquid was left in the mug at line 4?
  • Was the jacket lapel flipped before the reverse-shot coverage?

Between "Cut!" and "Moving on!", the script supervisor has a narrow 2-minute window while lighting and cameras reset. If a continuity defect escapes that window and the production wraps, the standing set is struck. A pickup day costs $18,000–$30,000 in crew labour alone for a lean 25–35 person crew (needacrew, 2026), and an average studio day runs to ~$500,000 (Careers in Film). Eyeline moves verification into the 2-minute window between takes, while the set is still standing.

Eyeline was built to solve this exact bottleneck: an autonomous, real-time continuity copilot that ingests camera takes side-by-side, deterministically isolates physical pixel discrepancies in sub-100ms, adjudicates intentional creative choices vs. accidental defects with Gemini 3.8 Flash, and generates synthetic B-roll cutaway pickups with Google Cloud Veo 3.1 if an unresolved defect is caught post-strike.

2. What Eyeline Does: The Strict 3-Pillar Architecture

Eyeline operates under a bounded 3-pillar ceiling engineered specifically to comply with hackathon Rule 7.B (strictly Google Cloud Gemini & Veo on Vertex AI, zero third-party off-the-shelf object detectors like YOLO or SAM):

🏛️ Pillar 1: Deterministic Alignment & Delta Isolation (Classical CV)

  • Zero Black-Box Detection: Uses classical computer vision under permissive licenses (opencv-python-headless, scikit-image).
  • Exposure & Geometry Normalization: Converts frames to YCrCb space for histogram and illumination equalization; extracts ORB features with RANSAC homography estimation to register camera perspectives.
  • Delta Masking & Contiguity Clustering: Employs multi-channel maximum difference masking and morphologic opening to isolate physical candidate regions while suppressing border edge-warp artifacts. Runs in ~25–85ms on CPU.

🧠 Pillar 2: Multimodal Continuity Adjudication (Google ADK & Gemini 3.8 Flash)

  • The Discriminative Filter: Classical difference masks are easily tripped by intentional variations (camera pans, focal length zooms, dramatic key light dims, actor performance timing).
  • Agentic Adjudication: Orchestrated via Google ADK (google.adk.agents.Agent) and Vertex AI gemini-3.8-flash. Receives high-resolution cropped candidate patches alongside take metadata, scene beat descriptions, and camera logs.
  • Empirical Impact: On our 16 paired negative control setups, Gemini retracted 4 out of 7 classical false alarms with 0 regressions, dropping the control false alarm rate from 43.8% down to 18.8% while retaining 100% of defect detections (15/16 recall).

🎬 Pillar 3: Generative Emergency Pickup Tool (Google Cloud Veo 3.1)

  • Post-Strike Safety Net: If an unscripted continuity break is discovered in the editing room after the physical set has been dismantled, Eyeline activates Pillar 3.
  • Veo 3.1 Synthesis: Generates 4.0-to-6.0-second broadcast-ready macro insert B-roll (e.g., wall clocks, establishing props, environmental inserts) tailored to the scene's color temperature and lighting.
  • Mandatory Disclosure Watermark: Every frame is burned with a visible, tamper-resistant SYNTHETIC CONTINUITY INSERT * VEO 3.1 watermark badge, providing ethical compliance and editorial clarity.

3. How We Built It & IBM Bob Provenance

Substantive modular components of Eyeline were authored under the IBM Partner Track using IBM Bob in headless execution mode. Transcripts for all 8 development tasks are permanently preserved in .bob-transcripts/ as verifiable proof of development provenance:

Task # Subsystem Cost Cap Actual Spend Provenance Transcript
1 Visual Diff UI Shell & Fixture Player 3.0 1.61 Bobcoins .bob-transcripts/task1-ui.json
2 Ground-Truth Schema & Pydantic Loader 2.0 1.56 Bobcoins .bob-transcripts/task2-schema.json
3 Classical CV Alignment & Diff Engine 4.0 2.03 Bobcoins .bob-transcripts/task3-vision.json
4 Pillar 2 Multimodal Adjudicator & Runner 3.5 2.46 Bobcoins .bob-transcripts/task4-adjudicator.json
5 Empirical Evaluation & Scoring Harness 2.0 1.61 Bobcoins .bob-transcripts/task5-scorer.json
6 Pillar 3 Veo Generative Cutaway Generator 4.0 2.14 Bobcoins .bob-transcripts/task6-veo.json
7 Benchmark Evaluation Runner CLI 2.5 1.55 Bobcoins .bob-transcripts/task7-runner.json
8 Anti-Circularity Forcing Functions 3.0 1.88 Bobcoins .bob-transcripts/task8-forcing-functions.json
TOTAL 24.0 14.84 Bobcoins 35.16 Bobcoins Remaining

4. Empirical Evaluation & Honest Scientific Metrics

We adhere to a strict Zero-Hallucination Policy: all benchmark numbers are generated directly from physical media evaluation via bench/run_benchmark.py and bench/sync_state.py.

The 32-Pair Benchmark Dataset

  • Balanced Protocol: 16 Defect Take Pairs + 16 Negative Control Pairs across 4 distinct scene templates (Diner, Office, Kitchen, Interrogation Room).
  • Physical Media: 64 rendered H.264 MP4 video clips (CRF 18) with realistic motion, grain, and lighting shifts.
  • Negative Control Rigor: Mandatory control pairs with intentional artistic changes (1.5-stop lighting dims, camera angle pans, focal zooms, dialogue pauses, color LUT grades) to measure discriminative power and prevent threshold gaming.

Measured Empirical Scoreboard

Metric Measured Value Formula Interpretation
Defect Detection Recall 15 / 16 = 93.8% $TP / P$ 15 of 16 seeded physical defects caught.
Spatial Localisation 6 / 16 = 37.5% $IoU \ge 0.3$ Classical CV isolates exact pixel-delta sliver.
Control FPR (Pillar 1 CV) 7 / 16 = 43.8% $FP / C$ Classical delta isolation on intentional camera/lighting changes.
Control FPR (Gemini 3.8 Adjudicated) 3 / 16 = 18.8% $FP_{\text{adj}} / C$ -25.0 pp reduction (4 false alarms retracted, 0 regressions).
False Passes (Escapes) 1 / 16 $FN = P - TP$ Escaped defect: pair_007 (hair/makeup line).

Statistical Disclosure (McNemar Paired Test)

Across the paired 16 negative controls evaluated before and after Gemini 3.8 Flash adjudication:

  • 4 controls retracted, 0 newly tripped (one-sided exact McNemar $p = 0.0625$, $n=16$).
  • The direction is unambiguous: Gemini fixed 4 false alarms and broke none. At $n=16$, the empirical 95% confidence interval is $[4.0\%, 45.6\%]$.

Spatial Localisation ($IoU \ge 0.3$) Explained

In 10 of 16 defect pairs, the detector candidate bounding box is 93.1% to 100% contained inside the ground-truth region (Inter / Pred_Area $\approx 1.0$). The standard IoU metric falls below 0.3 because ground-truth boxes annotate the entire semantic entity (e.g. full actor body for blocking or full face for makeup), whereas classical CV isolates the exact displaced edge or delta sliver ($\text{Area}{\text{delta}} \ll \text{Area}{\text{entity}}$).

5. Challenges We Overcame

  1. RANSAC Homography on Negative Controls: Camera angle pans and focal length zooms naturally produce perspective warp errors along frame boundaries. We engineered border-suppression margins and multi-band difference thresholds to prevent edge shear from creating artificial candidate clusters.
  2. Rule 7.B Permitted Model Ceiling: Because off-the-shelf object detectors (YOLO, GroundingDINO, SAM) are strictly barred, we relied entirely on classical computer vision for initial sub-100ms candidate bounding box generation, handing off cropped candidate regions to Gemini 3.8 Flash for semantic adjudication.
  3. Veo 3.1 Duration & Ingestion Handshake: The Veo 3.1 API enforces even duration constraints (duration_seconds in (4, 6, 8)). We authored a resilient SDK wrapper that downloads generated MP4 streams and burns the disclosure watermark frame-by-frame using OpenCV and ffmpeg.

6. Zero-Credential Offline Guarantee

Judges can inspect and test Eyeline with zero API keys and zero cloud configuration:

git clone https://github.com/helenkwok/eyeline.git
cd eyeline

# 1. Validate the 32-pair ground-truth benchmark
python3 -m bench.loader

# 2. Run the classical CV detector across all 64 real video clips
python3 -m bench.run_benchmark

# 3. Launch the On-Set Review Station & Judge Index
python3 -m http.server 8080 -d ui/
# Open http://localhost:8080/ in your browser
# Open http://localhost:8080/judge.html for the receipts and live presets

7. Links & Live Deployment

Built With

Share this project:

Updates

Submission history