Shadow Cut

"Multitasking is stretched to the limit" — Forbes. Shadow Cut tracks continuity in live pixels on set with Gemini & IBM, alerting directors before wrap to stop $50k reshoots.

Comment

Shadow Cut: The Director Still Directs. The Shadow Just Remembers.

Inspiration

I started with a simple question that wouldn't leave me alone: why does every AI filmmaking tool ignore the actual shoot?

Pre-production has AI screenwriters and pitch deck generators. Post-production has AI rotoscoping and automated color matching. But the moment the director calls "Action!"—when three cameras roll simultaneously, shooting ratios hit 30:1 to 150:1, and a single continuity error can cost $50,000+ to fix in reshoots—there is complete silence. The actual production phase has remained a multi-billion-dollar blind spot in a $25 billion market.

I spent weeks researching how film crews actually operate. Script supervisors aren't missing mistakes because of a lack of skill; they are drowning under a tsunami of data. In a 2022 Forbes interview, legendary Hollywood script supervisor Jayne-Anne Tenggren (Star Wars, James Bond, 1917) explained the crisis:

"With so much more information to notate, multitasking is stretched to the limit... Multi-camera challenges the balance between data entry and all the other tasks a script supervisor needs to perform."

When Scene 42 shoots on Day 1 and Scene 41 shoots on Day 18 on a different sound stage, human working memory breaks down. I realized the industry didn't need an AI that replaces people or pretends to automate art. It needed an invisible on-set companion—a "Shadow" that remembers every pixel the camera captured, alerting the director before the set is struck and pickups become impossible.

How I Built It: The 3-Tier Cascade & IBM Bob Architecture

Film sets cannot afford high latency or runaway compute bills. Running continuous multimodal video models on raw 4K feeds would cost thousands of dollars a day. To solve this, I designed a three-tier agentic cascade:

1. The Multi-Tier Pipeline

  • Tier 1 — Local YOLO-World (Every Frame, CPU/GPU, $0.00): Runs directly on the DIT (Digital Intermediate Technician) laptop. It extracts classes dynamically from the screenplay vocabulary, tracking bounding boxes and spatial coordinates at 50ms per frame.
  • Tier 2 — Gemini 3.5 Flash-Lite (Per-Take Anomalies, ~$0.002): When YOLO detects a state anomaly—or when scene ambiance requires validation—the proxy video and telemetry are sent via the Gemini Files API to Flash-Lite. Flash-Lite cross-references the scene's Plot Knowledge Graph to verify whether the anomaly is a real continuity problem or a false alarm.
  • Tier 3 — Gemini 3.1 Pro Preview (Deep Escalation, ~$0.10): Reserved for high-stakes, ambiguous, or non-linear narrative logic (e.g., "Did the character possess this prop before the setup in Scene 2?"). Triggered only when confidence is uncertain on a CRITICAL prop, capped at 50 calls/day.

2. IBM Bob as the Runtime Nervous System

IBM Bob was not used as a decorative chatbot or generic code assistant—Bob was the lead systems architect. Through iterative collaboration, Bob engineered:

  • 6 Dedicated Model Context Protocol (MCP) Servers: Exposing parse_script, analyze_take, check_continuity, flag_alert, query_memory, and generate_report.
  • 31 Strict Pydantic v2 Data Contracts: Eliminating untyped dictionary bugs across asynchronous pipeline boundaries.
  • 777-Line OpenAPI 3.0 Specification: (shadow_cut_orchestrate.yaml) engineered for zero-friction catalog ingestion into IBM watsonx Orchestrate.

3. Event Streaming & Production Interface

  • Confluent Cloud: A Kafka topic (shadow-cut.takes.uploaded) streams camera events in real time. If connectivity drops on location, a built-in FastAPI webhook queue seamlessly buffers events with zero data loss.
  • Next.js 14 Command Center: Deployed live on Google Cloud Run, featuring real-time confidence rings, A/B frame comparison sliders, screenplay deviation diff viewers, and an instant RAG memory chat.

The Economics: Why the Math Works

A standard feature film shoots roughly 100 takes per day across a 30-day shoot (~3,000 takes total). Using Flash-Lite for tier-two verification with targeted Pro escalations yields an economic equation that studio accountants can't ignore:

  • Cost per Take: $0.002 (Gemini 3.5 Flash-Lite)
  • Total Feature Compute: 3,000 takes × $0.002 + Rare Escalations ≈ $7.00 per movie
  • Prevented Reshoot Savings: $50,000 to $100,000+ (1 pickup day on location)
  • Return on Investment (ROI): 7,142x Return on Investment

The Bottom Line: For the price of two coffees from craft services on set, productions prevent a catastrophic, budget-breaking reshoot.

Who Buys Shadow Cut: The natural buyer is the production technology department of a streamer or mid-to-large independent studio (Netflix, A24, Apple TV+, HBO). The commercial model is a per-production license: a standard feature film shoot runs ~30 days, and at $150/shoot day — less than half a PA's day rate, roughly what a production spends on coffee from craft services — Shadow Cut costs $4,500 per film. Against a single $50,000 pickup day it prevents, that is an 11x return on the license fee alone, before counting the $7.00 in actual compute costs. The immediate pilot target is episodic television, where non-linear multi-camera schedules are the norm and the continuity problem is most acute: a show like The Bear or Succession shooting 8–10 episodes across 60+ days is the exact environment Shadow Cut was designed for.

Challenges I Faced

  1. The Holiday "No-Laptop" Constraint: At the start of the hackathon, I was away on holiday and didn't think to bring my laptop with me. At first, it felt daunting. But it turned out to be the best constraint possible: it forced me to plan, architect, and draft literally everything—the schemas, prompts, API contracts, edge case matrices, and voiceover scripts—from my phone. When I finally got back to my laptop, I wasn't wondering what to build; I was assembling a battle-tested blueprint.

  2. The Conda Dependency Quagmire: I originally planned to build and run the entire environment inside Anaconda. That turned into a nightmare. Conda's dependency solver choked repeatedly on the intersection of the newest google-genai SDK, Ultralytics YOLO-World, Pydantic v2 compiled binaries, and OpenCV headless builds. Channel conflicts and package mismatches stalled development for hours. I made the executive call to scrap Conda entirely, migrating to a lean, decoupled Python 3.11 virtual environment with strict pip pins. That solved the local environment, but the real test was deployment.

  3. The Google Cloud Run & SPA Routing Gauntlet: Deploying to Google Cloud Run was another serious gauntlet. First, OpenCV in Docker immediately crashed Cloud Run containers because minimal Linux images lack X11/OpenGL runtime C-libraries (libgl1, libglib2.0-0), requiring custom Dockerfile system package injections. Then, hosting the Next.js static SPA export directly inside FastAPI on Cloud Run triggered a routing war: FastAPI's StaticFiles catch-all route was aggressively swallowing /api/* and /health requests, while client-side SPA routing caused 404s on page refresh. I spent hours debugging until I engineered an explicit FileResponse fallback architecture that cleanly separates API routing from SPA client state.

  4. The "Zero Competition" Trap: Early on, I claimed "no automated continuity tools exist." When I researched deeper and found Studiovity, ScriptE, and Scriptation, I caught my own overclaim. Those tools track notes and paper logs; Shadow Cut tracks live pixels. Catching and correcting this made the architecture much sharper.

  5. Multiplicative Confidence Dampening: My initial math multiplied four decimal confidences together (0.8 × 0.7 × 0.8 × 0.9 = 0.40), artificially suppressing legitimate alerts. I refactored the engine so that narrative weight acts as a strict Decision Matrix gate, preserving high-confidence visual evidence.

  6. Zero-Hallucination Discipline: LLMs love to invent details that aren't on camera. I enforced an uncompromising policy: Shadow Cut only evaluates pixels physically inside the frame, never speculates about off-screen body parts, and grounds every claim in timestamped visual crops.

The Peak Experience: Why Real Footage Changed Everything

Early on, I considered using synthetic video data, generating AI test clips, or grabbing scenes from modern blockbusters. But synthetic data felt hollow, and modern films introduced messy copyright complications.

So I made a risky bet: I decided to test Shadow Cut on George A. Romero's 1968 classic, Night of the Living Dead (public domain).

I loaded the authentic 1968 shooting screenplay and fed in a 20-minute, 142-cut sequence of the farmhouse siege. It was terrifying—this was 58-year-old, grainy, black-and-white footage with erratic handheld camera work, heavy shadows, and antique props that modern models were never trained on. Not knowing if the pipeline was going to work on real footage was scary.

Then the system ran—and it became one of the greatest coding experiences I've had in years.

First, the pipeline caught Romero's famous set blunder: at 37:08, it flagged with 100% confidence the visible carpenter pencil handwriting ("UPPER RIGHT CORNER") on the barricade lumber facing the camera.

But what was truly breathtaking was what it caught next: two previously undocumented continuity errors that decades of film trivia buffs had overlooked:

  1. The Charcoal Lighter Fluid Shift: A spatial prop discontinuity where the lighter fluid container jumps position across reverse angles near the fireplace hearth before being touched.
  2. The Harry Cooper Key-Light Contrast Shift: A severe lighting continuity mismatch across cuts as Harry emerges from the cellar door.

The lighting finding was the real proof of the architecture: lighting is not an object. YOLO has zero concept of key-light falloff or shadow contrast because there is no bounding box for illumination. It was Gemini Flash-Lite’s multimodal reasoning that caught the directional shadow shift across cut transitions. That proved Shadow Cut wasn't just a bounding-box wrapper—it was genuinely understanding cinematic reality.

What I Learned & Deepening AI Collaboration

I already knew how to build with AI and use it as an engineering collaborator before this hackathon, but this project pushed that practice to an entirely new level.

Working with IBM Bob, Claude, and Gemini wasn't about outsourcing thought or generating boilerplate. It was an active brainstorming partnership. I used the models as intellectual sparring partners—testing architectural hypotheses, stress-testing Pydantic contracts, and simulating film set edge cases. Bob gave me the velocity to build 6 production MCP servers and a 777-line watsonx Orchestrate specification in days, but orchestrating the components into a cohesive, fault-tolerant system was where human intuition drove the ship. It was genuinely a lot of fun.

What's Next for Shadow Cut

  • Real-Time SDI/HDMI Hardware Capture: Building an edge box that plugs directly into camera video transmitters (Teradek) for instantaneous on-set analysis as the camera cuts.
  • Self-Serve Upload UI: A drag-and-drop take upload tab for the Command Center so producers can submit proxy clips directly from the browser. The backend /api/takes/upload endpoint already accepts multipart video — the constraint is architectural rather than developmental: Tier 1 (YOLO-World) is intentionally designed to run on local DIT hardware on set, not on Google Cloud Run, because film sets need sub-50ms frame analysis at $0.00 per frame. A browser upload bypasses Tier 1 and routes directly to Flash-Lite, which is exactly how Shadow Cut would operate for remote post-production review workflows.
  • Multi-Camera Temporal Alignment: Correlating simultaneous A/B/C camera takes to detect spatial continuity discrepancies across differing focal lengths.
  • Gemma 4 Fine-Tuning: Distilling continuity patterns into open-weights edge models that run completely offline in remote, zero-internet filming locations.

The director still directs. The Shadow just remembers.

Built With

Share this project:

Updates

Submission history