Project X : Agentic Script-to-Screen QA

An autonomous AI director that compares screenplays with generated video, detects cinematic mismatches, scores fidelity, and generates targeted fixes using Gemini, Google ADK, and ClickHouse.

Comment

Inspiration

AI video generation has become incredibly powerful, but filmmaking with it still has a frustrating bottleneck: verification.

A filmmaker can write a precise screenplay or generation prompt specifying dialogue, blocking, camera movement, lighting, sound, props, and continuity then receive a beautiful video that quietly ignores several of those instructions.

Checking every generated shot manually is slow, subjective, and becomes painful when a scene goes through multiple generations.

Project X was inspired by a simple question:

What if an AI director could watch the generated footage, read the screenplay, and tell you exactly where the model followed or failed to follow the director's intent?

Project X began with a SceneFlow-derived open-source script-to-screen synchronization foundation. For the hackathon, I transformed that manual review workflow into an agentic quality-control system for AI filmmaking.

What it does

Project X is an autonomous script-to-screen QA system for AI filmmakers.

A creator provides a screenplay and an AI-generated video. The Project X Director Agent then analyzes the production through a deterministic multi-step workflow:

  1. Parses the screenplay and understands dialogue, actions, camera directions, shots, audio, VFX, transitions, environments, and Project X's more technical Auteur Script staging directives.
  2. Analyzes the actual video with Gemini multimodally, grounding observations in the supplied footage rather than only the text prompt.
  3. Aligns screenplay beats to video timestamps automatically.
  4. Scores script-to-screen fidelity across eight cinematic dimensions:
    • Dialogue
    • Action / blocking
    • Camera
    • Shot composition
    • Audio
    • VFX
    • Environment
    • Transitions
  5. Detects critical failures such as a missing camera move, omitted prop interaction, incorrect framing, continuity drift, or mistimed audio.
  6. Lets the filmmaker jump directly to the failed moment in the synchronized screenplay/video interface.
  7. Generates targeted regeneration instructions instead of asking the filmmaker to rewrite the entire prompt.
  8. Stores generation intelligence in ClickHouse, allowing multiple attempts of the same scene to be compared over time.

Instead of simply saying that a generation is "bad," Project X can explain:

Expected: physical dolly-in toward the actor.
Observed: static medium shot.
Result: partial adherence.
Fix: explicitly require physical camera translation, prohibit digital zoom, preserve the subject's screen position, and define the final framing.

The goal is to turn prompt iteration from guesswork into a measurable production workflow.

How we built it

Project X combines a filmmaking-focused frontend with an agentic Google Cloud backend.

Google ADK + Gemini

The central component is the Project X Director Agent, implemented with the official @google/adk.

It exposes six specialized tools:

  • parse_script
  • analyze_video
  • evaluate_adherence
  • generate_regeneration_prompt
  • persist_analysis
  • query_generation_history

The workflow is intentionally deterministic because cinematic QA should be reproducible: screenplay understanding happens before video inspection, inspection happens before scoring, and scoring happens before recommendations or persistence.

For multimodal analysis, Gemini receives the actual video asset together with structured screenplay beats. Local MP4 data and supported Google Cloud Storage URIs can be passed into the analysis pipeline.

Vertex AI

Production mode is designed around Gemini 2.5 on Vertex AI, using Google Cloud Application Default Credentials rather than exposing model credentials in the browser.

This keeps the AI runtime server-side and allows the Cloud Run service identity to access Google Cloud resources securely.

ClickHouse Cloud

ClickHouse is the production-intelligence layer.

Project X stores both high-level generation runs and granular cue evaluations, including:

  • generation number
  • screenplay cue type
  • expected instruction
  • observed result
  • adherence score
  • status
  • confidence
  • failure reason
  • timestamps

This makes questions such as these possible:

  • Did Attempt 3 actually improve over Attempt 2?
  • Which cinematic category is consistently failing?
  • Is a model good at dialogue but poor at camera choreography?
  • Which prompt revisions improved continuity?

ClickHouse therefore becomes more than logging it becomes the historical memory of the filmmaking process.

Google Cloud Storage

Google Cloud Storage is used for persistent analysis artifacts and media-oriented workflow data. Large binary assets remain separate from analytical records while ClickHouse stores the intelligence around them.

Deployment architecture

The production architecture is built around:

ChatGPT Sites → Google Cloud Run → Google ADK / Vertex AI → Google Cloud Storage + ClickHouse Cloud

ChatGPT Sites hosts the filmmaker-facing experience, while Cloud Run provides the secure API and agent runtime.

Challenges we ran into

The hardest challenge was making sure Project X was actually analyzing footage rather than merely appearing to.

An early implementation sent screenplay information to Gemini but did not attach the video itself. That could produce convincing-looking analysis without grounding it in the footage. I replaced that path with genuine multimodal video attachment and added validation metadata to prove that the media asset was included.

Another challenge was graceful failure. AI demos often silently replace unavailable services with fake data. I wanted Project X to be transparent instead.

In live mode:

  • failed Gemini analysis becomes uncertain, not fabricated observations;
  • unavailable ClickHouse is shown as disconnected;
  • local artifact storage is not labeled as Google Cloud Storage;
  • benchmark/demo data is isolated from production behavior.

Cross-origin deployment was another challenge because the frontend and backend live on different platforms. The backend therefore needed controlled CORS, a centralized frontend API base URL, and a clean separation between public frontend configuration and server-only credentials.

Finally, integrating an existing open-source synchronization foundation responsibly meant being explicit about what existed before the hackathon and what was newly engineered. The repository preserves the upstream MIT attribution and documents the new Agentic Cinema components separately.

Accomplishments that we're proud of

I am especially proud that Project X evolved from a manual synchronization viewer into a real filmmaking QA pipeline without throwing away the workflow that already made the interface useful.

The hackathon implementation now includes:

  • an official Google ADK Director Agent
  • six registered agent tools
  • real Gemini multimodal video analysis
  • timestamp-grounded script-to-screen alignment
  • deterministic eight-category fidelity scoring
  • interactive critical-failure diagnosis
  • targeted regeneration prompts
  • ClickHouse cross-generation analytics
  • Google Cloud Storage artifact persistence
  • truthful runtime provenance indicators
  • a dedicated mismatch benchmark scene
  • production Cloud Run containerization
  • controlled cross-origin support for the ChatGPT Sites frontend

The current verification suite contains 38 passing tests across 10 suites, with TypeScript checks and both frontend/backend production builds passing locally.

More importantly, the system is designed so that a judge can distinguish a genuine cloud call from a demo fixture or disconnected service.

What we learned

The biggest lesson was that agentic does not have to mean unpredictable.

For creative quality control, a deterministic agent workflow can actually be more valuable than giving a model unlimited freedom. Every stage has a clear responsibility and can be inspected independently.

I also learned that multimodal filmmaking systems need two different forms of memory:

Artifact memory the actual scripts, videos, and analysis outputs.

Production intelligence — what happened across attempts, which instructions failed, and whether revisions improved the scene.

Google Cloud Storage and ClickHouse fit those two roles naturally.

Another important lesson was that provenance and transparency matter. An AI filmmaking tool should be able to tell creators not only what it thinks, but what footage it inspected, which service produced the analysis, and whether the historical data is real or a fallback.

What's next for Project X Agentic Script-to-Screen QA

The next step is to expand Project X from single-scene QA into a production-wide intelligence system.

Future directions include:

  • full-film continuity tracking across scenes
  • automatic character and prop state tracking
  • model-to-model benchmarking for Veo, Runway, Kling, Seedance, and other video generators
  • shot-level cost and iteration analytics
  • collaborative director / editor review
  • automatic regeneration execution after human approval
  • production-wide style drift detection
  • searchable historical filmmaking intelligence powered by ClickHouse

The long-term vision is for Project X to become an AI production supervisor for generative cinema:

not another tool that simply generates more footage, but an agent that helps filmmakers understand whether the footage they generated is actually the film they intended to make.

Built With

Share this project:

Updates

Submission history