CineQA-Agent

Inspiration

We started somewhere else entirely.

The original idea was a QA system for AI-generated video: decompose a director's prompt into atomic, checkable visual claims, verify each one against the generated frames, and automatically repair whatever failed. We built it. It worked, sort of.

Then we looked closely at why claims were failing, and found something that changed the whole project.

A shot kept failing the claim "the traveler moves from screen-left to screen-right." The video looked right — the character was clearly travelling. But the camera was tracking them, so in screen space the subject never moved at all. Background scrolled; subject stayed centered.

Neither the generator nor the verifier was wrong. The prompt had never said whether the camera holds or follows. Two models each made a reasonable guess, guessed differently, and our verification faithfully reported the disagreement as a defect.

We kept finding the same shape of problem. Our claim extractor invented details — a door lock, a bolt, an arched hallway — that the director's prompt never mentioned, then dutifully failed the video for not containing them. Again: not a bug in verification. A vacuum in the prompt, filled independently by two different models.

That reframed everything:

The bottleneck in AI filmmaking isn't that the model guesses badly. It's that you never told it what you meant, so it had to guess.

So we rebuilt the product around the upstream problem: an agent that interviews you like a cinematographer before a single frame is generated.

What it does

You describe a shot in one loose sentence.
The agent interviews you, one question at a time, across the dimensions that actually determine whether a generated shot succeeds — framing and camera motion, action and blocking, location, lighting, style. It asks only about what you left ambiguous, and each question's options are conditioned on your previous answers, so it never offers a camera move that contradicts one you already chose.
It has an opinion. Every question marks one option as recommended, with a director's note explaining what that choice does to the audience — not what equipment it implies.
It decides what to ask first by querying its own history in ClickHouse over MCP: which dimensions have users most often had to fix after the fact? Ask those first.
You confirm a keyframe for about a cent before spending on video generation. The frame you pick becomes the video's opening frame.
It generates with Gemini Omni, then verifies the result and — critically — does not fix anything itself. It tells you what it saw and hands you a one-sentence instruction you can paste into conversational editing. You decide.

How we built it

CineQA-Agent is built on an agentic, observability-driven closed-loop architecture consisting of four specialized modules:

              YOUR ONE-LINE IDEA
                      │
                      ▼
   ┌───────────────────────────────┐         ┌────────────────────────┐
   │  Prompt Director Agent        │◀──MCP───│  ClickHouse            │
   │  adaptive interview           │  query  │  guidance_events       │
   │  · asks only what's ambiguous │         │  (what users regretted │
   │  · order decided by history   │──logs──▶│   in past sessions)    │
   └───────────────┬───────────────┘         └────────────────────────┘
                   │
                   ▼
      Keyframe check (~1¢)  ──▶  you confirm the frame
                   │
                   ▼
             Gemini Omni  ──▶  TAKE 1
                   │
                   ▼
   ┌───────────────────────────────┐         ┌────────────────────────┐
   │  Verification Engine          │──logs──▶│  ClickHouse            │
   │  + split-conformal layer      │         │  verification_ledger   │
   └───────────────┬───────────────┘         └────────────────────────┘
                   │
         ┌─────────┴──────────┐   tiered trust: measured per claim type
         ▼                    ▼
   ≥80% precision       <80% precision
   autonomous verdict   → director review queue
   colour · direction   action · relative size
   position · state     camera · micro-topology
         │                    │
         └─────────┬──────────┘
                   ▼   flags raised — nothing is auto-fixed
        Observation Report
        "4.3s–7.0s · the door reappears intact"
                   │
                   ▼
        ★ YOU write what to change, in your own words
                   │
                   ▼
        Prompt Optimizer  ──▶  precise instruction,
                   │           preserving what you didn't ask to change
                   ▼
        Omni incremental edit (previous_interaction_id)  ──▶  TAKE 2

Prompt Director & Adaptive Interview Agent:
- Interviews the user one dimension at a time — framing and camera motion, action and blocking, location, lighting, style — with each question's options conditioned on every answer given so far.
- Before proposing its first question, it queries its own history in ClickHouse over MCP: which dimensions have users most often had to fix after the fact? That signal, not a fixed script, decides what gets asked first.
- Generates high-fidelity keyframe candidates (image seed frames) so the user can confirm the shot for about a cent before spending on full video generation.
Multimodal Physics & Temporal Verification Engine:
- Employs Gemini Multimodal Vision to analyze spatio-temporal video slices.
- Verifies Scene Consistency, Photometric Lighting Stability, and Kinematic Plausibility (gravitational acceleration, collision response, continuity).
Rigorous Uncertainty Quantification via Split-Conformal Prediction:
- Instead of trusting raw LLM heuristics, we calibrate visual verification scores using distribution-free Split-Conformal Prediction.
- For confidence level 1 − α (where α = 0.10 for 90% statistical confidence), we compute non-conformity scores sᵢ = 1 − p̂(yᵢ) over a calibration set {(X₁, Y₁), …, (Xₙ, Yₙ)}.
- The conformal critical threshold q̂ is computed as: $$\hat{q} = \text{Quantile}\left( {s_i}_{i=1}^{n},\; \frac{\lceil (n+1)(1-\alpha) \rceil}{n} \right)$$
- If the predicted non-conformity exceeds q̂, CineQA automatically flags high uncertainty and flags the claim for human review or surgical compensation.
Tiered Trust — the system knows which of its own judgments to trust:
- Not all visual claims are equally verifiable, so we measured per-claim-type precision across 92 labelled shots and let the data decide what the system is allowed to rule on alone.
- Above our 80% autonomy bar — colour (100%), direction (93%), relative position (92%), object state (91%) — verdicts are reported as autonomous verified decisions.
- Below it — action (57%), relative size (67%), camera aesthetics, micro-topology — CineQA refuses to issue a binary verdict at all. It extracts objective coordinates and kinematic signals, and routes the claim to a director review queue instead.
- This is the conformal principle applied one level up: a system reporting 82.6% overall accuracy as a single number is less useful than one that tells you which 17% it cannot be trusted on.
Guided Observation & User-Directed Remediation:
- When a defect is found, the system does not fix it automatically. It surfaces a human-readable observation: the timestamp, the location in frame, and what's inconsistent.
- The user decides what to change and writes it in their own words. The system then rewrites that into the precise, technical language the model actually responds to — explicitly preserving everything the user didn't ask to change.
- That refined instruction is sent to Gemini Omni via previous_interaction_id, which enables true incremental editing: the model remembers the previous take and changes only what was asked.
Production Telemetry & Observability:
- Every guidance event (questions shown, chosen, or tweaked) and every verification verdict, per claim, per take, is logged in real-time to ClickHouse Cloud as two tables — guidance_events and verification_ledger — the same store the Prompt Director queries live over MCP.

Challenges we ran into

The "Morphing" vs. Camera Movement Dilemma: In generative video, fixing both the first and last frame (I2I interpolation) often locked the 2D vanishing points, causing AI models to morph objects instead of moving the 3D camera. We designed an adaptive heuristic: when intentional camera sweeps (pan/tilt/truck) are detected, CineQA dynamically unlocks the end-frame constraint to allow natural camera motion.
Deterministic Claim Scoring: Video hallucinations are often subjective. By pairing deterministic OpenCV kinematic optical-flow measurements with LLM reasoning via JSON schema enforcement, we turned abstract cinematic flaws into quantified telemetry.
Resilient Vertex AI Authentication under Network Instability: Direct REST interactions with cutting-edge Omni preview APIs under fluctuating network conditions led to transient connection timeouts. We built an in-memory token-caching layer with exponential backoff and dual-channel gcloud CLI fallback to ensure uninterrupted generation.
Veo 3 will not let you turn off prompt enhancement: We spent a long time on surgical, token-level prompt repair before discovering — via a raw API error, "Veo 3 prompt enhancement cannot be disabled" — that our carefully engineered wording was being rewritten by another model before it ever reached the generator. This was one of several reasons we moved to Gemini Omni, where our elicited precision survives, and where previous_interaction_id enables true incremental editing: the model remembers the previous version and changes only what you asked for.

Accomplishments that we're proud of

An agent that declines to act on weak evidence.: Getting a model to follow a strong signal is easy. We seeded three regret events, and the agent read them, recognised the sample was too small, and kept its default question order anyway — then with six events it moved lighting to the first question and cited the history in its own explanation to the user. A system that knows when its own data isn't good enough yet is a more useful collaborator than one that always has an answer.
A learning loop you can watch change behaviour in real time.: Same prompt, same code, different database state, different first question. It takes about fifteen seconds to demonstrate and requires no hand-waving: the agent queries ClickHouse over MCP, the query result is visible, and the resulting question is visibly different. The regret metric is also self-balancing by construction — once a dimension gets asked about, tweaks to it stop counting as regrets, so no single dimension can monopolise the top slot.
Two independent checks caught the same real defect. : In one shot, our Tier-0 physics claim and a separate claim about the door's appearance each independently reported that a hole kicked through a wooden door sealed itself as the door swung open — a topological continuity failure that is genuinely hard to catch by eye. Two separate API calls, no shared state, same finding. That cross-validation is the strongest evidence we have that the verification is reading the video rather than pattern-matching the prompt.
We grounded the craft in actual craft. : The option vocabulary isn't invented — it comes from real cinematography references, with every technique annotated by its effect on the audience so the agent can match technique to intent. And we had the question ordering wrong at first: we asked about camera before action. Professional practice is the reverse — blocking determines the shot list, the lens, and the lighting positions — and the multi-agent previsualization literature independently orders it the same way. Fixing that made the conditioning between questions dramatically more coherent.

What we learned

VLM verification of physics and continuity is genuinely hard, and the literature says so. : VideoPhy-2 reports the best model reaching only 22% joint performance on challenging physical-commonsense scenarios, with conservation laws the most common failure. VideoZeroBench found that when models must both answer correctly and localize spatio-temporally, no model exceeds 1% accuracy. We stopped treating our verification failures as bugs to be engineered away and started treating them as a capability ceiling to design around.
A wrong verdict and a wrong suggestion cost completely different amounts.: This is the insight that produced our final architecture. An autonomous fix that's wrong wastes a generation, corrupts the ledger, and embarrasses you on stage. A suggestion that's wrong costs a glance and a decision not to use it. Reframing verification from judge to advisor lowered the accuracy bar by an order of magnitude while making the product better.
Naive heuristics over LLM output fail in ways that look like model failure. : We added a keyword safeguard so that a model describing a defect couldn't also return a passing verdict. Then every claim started failing. The cause: models describe passing checks by negating each item — "No morphing, no clipping, not floating" — and a substring match cannot see negation. Worse, our first fix (checking a 40-character window before each keyword) still failed on list-style negations, where "No" sits 65 characters from the last item. We eventually deleted the mechanism entirely rather than keep patching a heuristic that couldn't represent the thing it was judging.
One unscoped rule can have unlimited blast radius. : A global physics-sanity check applied to every claim meant that any artifact anywhere in the frame failed every unrelated claim — including one about lighting quality that was rejected because a boot vanished elsewhere in the shot. Since AI-generated video essentially always contains some artifact somewhere, this mathematically guaranteed near-total failure. Scoping the check to the claim's own subject fixed it.

What's next for CineQA-Agent

Audio-Visual Latent Synchronization: Extending verification to audio cues, foley physics, and lip-sync alignment.
Real-Time Mask-Guided Inpainting: Inpainting localized physics violations directly inside the latent space without re-rendering the full scene.
Director Style Transfer Kits: Allowing studios to fine-tune CineQA's inspection rules to their own unique cinematic lookbooks and grading guidelines.