SlateSync

Turning costly reshoots into instant on-set pickups: Multimodal continuity auditing and real-time schedule recovery.

Comment

Inspiration

Continuity errors are one of the most persistent, expensive problems in film production — a coffee cup that jumps from the right hand to the left, sunglasses that vanish and reappear, a prop on the wrong side of a table. Today a human script supervisor catches these by watching every take and cross-referencing notes and the script by memory, in real time, on set, under pressure. We wanted to know: could an agent do the same job — watch the footage, decide for itself what needs checking, pull the actual historical record, and reason about whether a change is a story beat or a mistake — instead of just running a fixed checklist?

What it does

SlateSync is an autonomous continuity supervisor. You give it a take (video), the scene and take IDs, and a screenplay excerpt. From there the agent runs the whole audit itself:

  1. Watches the take with Gemini's multimodal understanding and records what it sees — props, wardrobe, hand/object relationships, anything continuity-relevant — as structured observations.
  2. Decides on its own which of those observations are actually uncertain enough to need historical evidence, then queries a ClickHouse database of prior takes for that exact scene.
  3. Checks the story, not just the props: for anything that changed from history, it reads the screenplay excerpt and queries a script_events table to judge whether the change is explained by the narrative or is a real error.
  4. Delivers a verdict — PASS, REVIEW, CONTINUITY_RISK, or HIGH_PRIORITY_CONTINUITY_RISK — with the specific entity and attribute involved, the historical vs. observed values, which prior takes support that history, and a concrete recommended action (e.g. "capture a pickup before the kitchen set is struck").

The frontend shows all of this as a real production console: slate input on the left, video preview and a live tool-activity trace in the center, an evidence panel with the raw ClickHouse results on the right, and a color-coded decision banner across the bottom.

Critically, none of the pipeline is hard-coded. We never wrote "if a prop changes, check history." The agent decides when it's unsure and needs a query, decides when a change is explained, and closes out the case itself — Gemini's native function-calling drives every step.

How we built it

  • Agent core: Google's Agent Development Kit (google-adk) running an LlmAgent backed by Gemini, with four typed tools it calls at its own discretion: record_continuity_observations, historical-state querying, record_narrative_assessment, and submit_continuity_decision.
  • Historical evidence: ClickHouse, wired in as a real MCP server (mcp-clickhouse) via ADK's McpToolset — the agent gets list_databases, list_tables, and run_query directly, so it writes its own SQL rather than calling a wrapper function we designed in advance.
  • Data model: a continuity_entities table (scene, take, entity, attribute, value, confidence), a script_events table, and a takes table — seeded with a deterministic demo scenario: a coffee cup that stays in the right hand across five straight takes of Scene 12, then switches to the left hand on Take 26 with nothing in the script to explain it.
  • Backend: FastAPI, exposing a single /audit endpoint that returns the full extraction, every tool call with its paired response (for the activity trace), narrative assessments, and the final decision.
  • Frontend: Next.js (App Router) and React, a three-column console with a defensive evidence renderer that handles ClickHouse's raw result serialization safely.
  • Deploy target: Docker for both services, Cloud Run for hosting, Secret Manager for the Gemini and ClickHouse credentials, with the backend deployed first so its URL can be baked into the frontend's build.

We built it in phases, each one adding exactly one capability to the agent's tool list and re-verifying the whole system before moving on — extraction only, then historical lookup, then narrative reasoning, then the decision tool, then the real UI, then a live end-to-end smoke test, then containerization.

Challenges we ran into

  • Legacy ClickHouse in the dev sandbox. Our local environment only had a 2018-era ClickHouse available via apt, which doesn't understand a session setting the modern clickhouse-connect client sends by default (Unknown identifier: readonly). We confirmed the schema and seed pipeline both work correctly against it directly, isolated the exact failing setting, and documented that a real deploy needs a modern ClickHouse (e.g. ClickHouse Cloud) rather than papering over the gap.
  • A real dependency conflict, not a hypothetical one. google-adk pins mcp<2, while the official mcp-clickhouse package pulls in mcp>=2. Running it as an isolated subprocess via uvx — rather than as a direct backend dependency — sidesteps the conflict entirely and doubles as good deploy hygiene (a cold-start container shouldn't depend on PyPI being reachable).
  • Faking progress vs. showing real progress. /audit is a single blocking call, so we deliberately did not fabricate an incrementally-streaming activity log. The console shows a loading state, then the actual, complete tool-call trace once it's back — real streaming would need SSE/websockets, which we scoped out rather than fake.
  • Network egress in the sandbox blocked calls to Gemini's File API and generativelanguage.googleapis.com directly, so several of the "real" end-to-end runs could only be verified up to the point the network allowed, with the rest verified structurally (unit tests, mocked event streams shaped exactly like real ones, and a live smoke-test script ready to run the moment real credentials and network access are available).
  • Sandbox limitations meant no Docker or gcloud were available to us either. We verified everything short of an actual container build and Cloud Run deploy — including proving that the frontend's build-time API URL is genuinely baked into both the client and server bundles by building it with a fake URL and grepping the output — and left a precise, reviewed (not just assumed) command sequence for the real deploy.

What we learned

  • Letting the model decide when to query, rather than scripting a fixed check-every-observation pipeline, produces a genuinely more useful trace — you can see the agent recognize the one thing that's actually ambiguous instead of drowning the evidence panel in irrelevant lookups.
  • MCP is a clean way to hand an agent real query power without writing and maintaining a bespoke wrapper API for every table — and running it as an isolated subprocess is a good default for dependency isolation, not just a workaround.
  • "Verified" needs an honest, narrower definition when your sandbox is missing a piece of infrastructure (Docker, gcloud, a modern ClickHouse, network egress to the model API). Being explicit about exactly what was and wasn't run — and why — is more useful, and more trustworthy, than claiming end-to-end success you can't actually back up.

What's next for SlateSync

  • A real live Gemini + ClickHouse Cloud end-to-end run against actual dailies footage, using the smoke test we built specifically to gate that.
  • Real streaming of the activity log via SSE, so the console shows tool calls as they happen instead of after the fact.
  • Broader continuity coverage beyond hand/object and wardrobe state — lighting continuity, background/extras continuity, and camera/lens metadata.
  • A persistent per-production dashboard so a script supervisor can review flagged risks across an entire shoot day, not just one take at a time.

Built With

Share this project:

Updates

Submission history