CineMatch AI

A Gemini-powered multi-agent system using parallel tool calling to analyze script-to-screen continuity and live production data.

Comment

Inspiration

Every film production loses money to the same invisible enemy: continuity drift. A coffee cup that switches hands between takes. A jacket that's buttoned in the wide shot and unbuttoned in the close-up. A prop that quietly vanishes from a shelf three scenes later. On a real set, this is caught by a script supervisor flipping through binders — if it's caught at all. Reshoots for continuity errors routinely cost productions tens of thousands of dollars per day.

We kept coming back to the same question: what if a set had a team of tireless AI agents watching every frame, cross-referencing every script page, and flagging drift before it ever reached the editing room? Not a single chatbot bolted onto a form, but a genuine multi-agent system that could reason, query a live production database, and even pre-visualize a shot before the crew rolled camera. That question became CineMatch AI.

What it does

CineMatch AI is an orchestrated multi-agent production intelligence suite built on Google's Agent Development Kit and Gemini. A CineMatch_Master_Agent delegates work to specialist sub-agents, each armed with real, callable tools:

Continuity Agent – cross-references script pages against footage to build a continuity graph and flag drift in props, wardrobe, and blocking. Production Planning Agent – runs a multi-turn tool-calling loop against a live Firestore database, deciding for itself when it needs to query cast, equipment, or scene data before answering a logistics question. Safety Agent – pulls real Google Maps data to evaluate filming locations and crew amenities. Video Generation Agent – translates a scene description into an optimized Veo prompt and renders an 8-second pre-vis clip with synchronized audio. Podcast Generator – synthesizes script and continuity notes into a two-speaker recap using distinct Gemini-generated voices.

Every agent's output feeds a shared Command Center: a live agent activity feed, aggregated findings, pending decisions, and production alerts — giving a first AD or script supervisor a view of the whole shoot's health at a glance.

How we built it

The system is orchestrated with Google's Agent Development Kit (@google/adk), using an LlmAgent master agent that delegates to specialist sub-agents running on Vertex AI Gemini (Pro for orchestration, Flash for specialist reasoning). Agents call tools through a Model-Context-Protocol-style function schema — mcp_query_vault lets any agent autonomously query live Firestore collections (scenes, cast, equipment, assets), and searchLocations gives the Safety Agent access to Google Maps data.

For generative media, we integrated Veo for pre-visualization video with native audio, Lyria for original score generation, and the Gemini Live API over a WebSocket relay for a real-time, bidirectional voice assistant a director can talk to on set. The backend runs on Node.js 22 and Express 5, with Multer handling media uploads and ws powering the live voice connection. Firestore serves as the production vault, and Google Cloud Storage holds footage, storyboards, and generated media.

The whole thing is cloud-native by default: a Docker image (node:22-slim) with OpenTelemetry pre-loaded, a Cloud Build pipeline that builds, pushes to Artifact Registry, and deploys to Cloud Run in one shot, and secrets (Gemini, Google Maps, Resend API keys) injected from Secret Manager at deploy time rather than hardcoded. A Knative service spec declares autoscaling behavior (max 5 instances) for the Cloud Run deployment.

Challenges we ran into

Getting agents to genuinely decide when to call a tool — rather than always calling it, or never calling it — took real iteration on the Production Planning Agent's prompting and loop structure. We also had to build guardrails around every generation call: configurable Vertex AI safety settings across harassment, hate speech, sexual content, and dangerous content, with a graceful fallback that surfaces a CRITICAL_REVIEW_REQUIRED flag instead of silently failing or crashing the request — something we felt was non-negotiable for anything positioned as an enterprise studio tool. Wiring a real-time bidirectional voice agent through a WebSocket relay to the Gemini Live API also meant handling audio streaming and connection state in a way a typical request/response API integration never has to.

Accomplishments that we're proud of

We're proud that CineMatch isn't a single prompt wearing different hats — it's a real orchestrator-to-specialist hierarchy where each agent has its own model, its own tools, and its own job. The Production Planning Agent in particular proves genuine MCP-style tool-calling and live database grounding: it queries Firestore autonomously and produces a deterministic, hallucination-free answer instead of a plausible-sounding guess. We're also proud of how far the generative media stack reaches in one platform — from Veo pre-vis clips with synchronized audio, to Lyria-composed score cues, to multi-speaker podcast recaps — all orchestrated rather than hand-triggered one at a time. And we shipped it as an actual deployable cloud service, not a local demo: Dockerized, built and deployed through Cloud Build, with secrets properly managed through Secret Manager.

What we learned

Building this reinforced how much of "agentic" behavior lives in the tool-calling loop, not the model itself — the same Gemini model produces a vague answer or a grounded, verifiable one depending entirely on whether it's been given the right tools and the discipline to use them. We also learned a lot about designing for graceful degradation: an enterprise-facing creative tool has to fail safely and visibly (a CRITICAL_REVIEW_REQUIRED flag) rather than fail silently. And working across Gemini, Veo, Lyria, and the Live API in one project gave us a much clearer picture of how Google's generative media stack fits together end-to-end, from text and vision reasoning through to real-time voice and generated video.

What's next for CineMatch AI

Next, we want to close the loop from detection to action: automatically generating a reshoot list ranked by cost and schedule impact whenever the Continuity Agent flags drift. We'd also like to give the Character State Tracking and Scene Spatial Memory agents persistent, queryable timelines that a script supervisor can scrub through like a video editor, rather than just a stream of alerts. On the infrastructure side, we're looking at tightening the multi-user studio access story — building out the bcryptjs/jsonwebtoken auth scaffolding into full role-based permissions so a director, script supervisor, and studio exec can each see a view of the Command Center suited to their role.

Built With

Share this project:

Updates

Submission history