-
-
Screenshot 4
-
Screenshot 2
-
Screenshot 3
-
Screenshot 1
Inspiration
On every film set, a script supervisor has one brutal job: catch every continuity error across dozens of takes — did the actor's jacket stay zipped, did a prop stay the same color, did it stay in the same hand. It's done almost entirely by memory and Polaroids, and a missed error can mean an expensive reshoot or a VFX fix discovered only in the edit. We wanted to know: could an AI agent actually do this job well — not as a toy demo, but as something a real script supervisor could work through during an actual shoot day?
What it does
Continuity Guardian ingests raw take footage, uses Gemini's multimodal video understanding to extract structured continuity data for every take — props, their color, costume state, actor positioning, camera notes — and automatically flags inconsistencies between takes of the same scene. All of that data lives in ClickHouse, queried live by a Gemini/ADK agent through the official ClickHouse MCP server, so a user can ask plain-English questions like "what continuity issues are flagged for SC01?" and get an answer grounded in real data. The dashboard supports a full working review process, not a one-shot demo:
- Scenes Overview — browse every scene with open-flag counts at a glance
- Real video playback per take — click a thumbnail to watch the actual footage in a modal player, not just a static frame
- Severity-graded flags (high / medium / low) that actually mean something — a color or state change ranks differently than a subtle position shift
- Missing-object detection — catches a prop that vanishes for a take and reappears, even when its position matches before and after
- A resolve workflow — mark flags resolved as you work through them across a shoot day
- Natural-language chat, grounded live in ClickHouse via the ADK agent and MCP ## How we built it Ingestion: google-genai, calling Gemini with a structured-JSON prompt against each take's raw video, extracting objects, actor/costume notes, camera notes, and risk flags — with explicit rules for reporting the final on-screen state (not a mid-action snapshot), body/landmark-relative positioning (so camera angle changes don't create false positives), and always naming color, since color changes are some of the most common real continuity errors. Storage & analytics: ClickHouse Cloud, with a schema purpose-built for this — shots, detections, continuity_flags, camera_notes — including the take's own video and thumbnail, queried both by our Python loader (write path) and by the agent via the official mcp-clickhouse MCP server (read path). Agent: Google's Agent Development Kit (ADK), with the ClickHouse MCP server wired in as a live tool via MCPToolset/StdioConnectionParams — the agent genuinely queries live data at runtime. Continuity detection: a fuzzy cross-take matching algorithm that clusters detections into the same real-world object across takes even when Gemini describes them slightly differently each call, with an explicit color-priority check so a genuine color change is never diluted by shared incidental words (like both takes mentioning "silver blades"). Dashboard: FastAPI + a from-scratch HTML/CSS/JS frontend, deployed on Render, database on ClickHouse Cloud, with a shared-secret access gate on the Gemini-quota-costly endpoints. ## Challenges we ran into The hardest problem wasn't the AI pipeline itself — it was matching the same real-world object across separate Gemini analysis calls. Gemini rarely describes an object with identical wording twice ("small outlined blue circle" vs. "Outlined blue circle"), so naive exact-match comparison silently missed real continuity errors. We built a fuzzy-matching layer, tested it against real footage, and found — and fixed — two real regressions along the way: word-form drift ("handles" vs "handled") wrongly triggering false positives, and later, a genuine color change (green vs. yellow scissors) getting masked because shared incidental words ("silver,""metal,""blades") diluted the similarity score. We fixed this with an explicit color-priority check that treats a genuinely conflicting color word as an override signal, verified against every real example collected during testing. We also found a genuine, still-open limitation: one of our real test takes includes an on-screen prop swap — scissors visibly changing from yellow to green mid-take — and Gemini's analysis inconsistently reports which color is the true final state, even after two rounds of prompt refinement. This is an honest limit of single-pass video understanding, not a bug we chose not to fix. On the infrastructure side, we hit a real dependency-compatibility bug: an unpinned mcp package version could silently break the ADK agent on a fresh deploy, traced down to ADK's own declared version constraint and fixed. We also navigated a genuine platform-wide Render outage during development, and ClickHouse Cloud's free-tier idle/wake behavior, both of which taught us to design the agent's MCP connection with a realistic timeout rather than the 5-second default. ## Accomplishments that we're proud of ● A genuinely live ClickHouse MCP integration — the agent's answers are grounded in real-time queries, not a decorative mention ● Finding, diagnosing down to the exact string comparison, and properly fixing two real false-positive/false-negative bugs in our own continuity-matching logic, rather than shipping something that merely looked correct in a demo ● Real in-browser video playback per take, not just a static screenshot ● A dashboard that supports an actual review workflow — browse, watch, review, resolve, ask — not a single unrepeatable demo path ## What we learned Multimodal video understanding is powerful but not infallible — the same clip analyzed carefully frame-by-frame can reveal that the model's own reported "final state" doesn't match what's actually on screen. Testing against real, sometimes messy footage (rather than trusting a synthetic example) was what actually surfaced every real bug we fixed — including two rounds of regressions in our own comparison logic that a synthetic test alone would never have caught. ## What's next for Continuity Guardian ● Lighting and color-grade continuity — extending detection beyond objects and costume to visual grade consistency across takes ● Full dailies-reel ingestion — moving from single takes to processing an entire reel of raw footage in one pass ● Multi-user review workflows — letting a director, editor, and script supervisor collaborate on the same flagged issues ● Object re-identification via visual embeddings rather than text descriptions, to close the remaining gap in fast on-screen prop swaps
Log in or sign up for Devpost to join the conversation.