CineAudit Sentinel

Autonomous DIT & Dailies QA engine. Transforms raw footage into ClickHouse vector telemetry, audited by Gemini 2.5 Flash to calculate VFX vs reshoot costs and prevent $70k set errors.

Comment

"In high-end film production, AI shouldn't just generate synthetic content; it must protect the studio's physical capital. CineAudit Sentinel is not a chatbot. It is a deterministic, agentic quality-assurance engine built specifically for the ruthless economics of Hollywood."

🎬 The Inspiration: The $70,000 Boom Mic & Human Fatigue

Picture this: It's 3:00 AM on a massive virtual production stage in London. The crew is on hour 14 of a grueling shoot day. The Director calls "Cut. That's a wrap for this location." The crew immediately begins striking the lighting rigs, dismantling the set pieces, and wrapping the equipment.

The next morning in the editing bay, the post-production team realizes a catastrophic continuity error: The lead actor switched their coffee mug to the opposite hand between Take 01 and Take 02, and a boom microphone dropped into the top-right corner of the frame for exactly 1.2 seconds.

Because the physical set has already been dismantled, the studio is forced into an emergency reshoot. The logistical cost of recalling the union crew, A-list talent, and securing the soundstage? Between $35,000 and $70,000 per stage hold.

We realized that modern film sets lack autonomous, real-time forensic auditing. Human eyes—even those of elite Script Supervisors and DPs—experience fatigue. We didn't want to build an LLM wrapper that just "summarizes video." We set out to build an industrial-grade Digital Intermediate (DIT) autonomous agent. An engine capable of translating pixel anomalies and acoustic variances into hard USD financial metrics, empowering the 1st Assistant Director to confidently authorize a set strike.

🚀 What it does: The Hollywood DIT Suite

CineAudit Sentinel is a serverless, multi-agent architecture that ingests raw camera footage (Dailies), transforms it into high-velocity telemetry, and executes a complex financial calculus using Agentic AI to prevent continuity disasters before the crew leaves the set.

It provides four studio-grade forensic modules accessible via a DaVinci Resolve-inspired interface:

1. ◨ Interactive Split-Wipe & Difference Matte Forensic Suite

Operators cannot rely on playing two videos side-by-side. We built a GPU-accelerated HTML5 UI that allows DITs to scrub through a dynamic split-wipe bar. For deeper analysis, the Difference Matte mode subtracts pixel matrices in real-time, completely blacking out matching pixels and illuminating photometric deviations (like a swapped prop or underexposed lighting) in Neon Magenta.

2. ⛶ Spatial Grounding Viewfinder (Gemini Vision)

When an anomaly is flagged, Gemini 2.5 Flash acts as an AI script supervisor. It doesn't just describe the error textually; it performs exact spatial grounding, drawing production-ready bounding boxes (styled like ARRI/RED camera viewfinders) over the exact coordinates of the boom mic or misaligned prop.

3. 🎨 Native ASC-CDL Match Grade Engine

If Take 02 is underexposed compared to the Master Take 01, our system extracts the luminance deltas from ClickHouse and calculates the exact American Society of Cinematographers (ASC-CDL v1.01) mathematical nodes (Slope, Offset, Power). With one click, the system generates a native .cdl XML file that colorists can drag-and-drop into DaVinci Resolve to instantly match exposure without manual color wheels.

4. 🎚️ 48kHz Acoustic Room Tone Telemetry

Visual continuity is only half the battle; an undetected shift in background room tone (e.g., a generator turning off) destroys dialogue editing. We extract 16-bit PCM audio at 48,000 Hz, calculating Frame-accurate acoustic energy (RMS) and instantly flagging decibel shifts on a segmented digital VU meter.

⚙️ How we built it: The ClickHouse & Gemini Engine

To achieve sub-15 millisecond latency on video analysis, we had to stop treating video as a static blob and start treating it as time-series vector telemetry. We orchestrated two massive tech pillars deployed via Google Cloud Run (Scale-to-Zero for maximum cost-efficiency during non-shooting days):

Pillar 1: ClickHouse Cloud (The Telemetry Heart)

We built a custom Python ingestion pipeline using OpenCV. Instead of sending full video files to an LLM, we sample the footage, extract 64-dimensional $L_2$ normalized spatial signatures, and ingest them directly into ClickHouse MergeTree tables.

ClickHouse serves as our ultra-fast vector engine, executing cosine distance queries to pinpoint the exact timecodes of anomalies across thousands of frames in milliseconds:

$$CosineDistance(u,v) = 1 - \frac{u \cdot v}{|u|_2 |v|_2}$$

Simultaneously, we extract audio and store acoustic energy (RMS) telemetry directly alongside the visual vectors in ClickHouse:

$$RMS_{dBFS} = 20 \log_{10}(\max(\sqrt{\frac{1}{N} \sum s_i^2}, 10^{-5}))$$

Pillar 2: Google Cloud Vertex AI (The Agentic Brain)

Once ClickHouse surfaces the mathematical discrepancies, Gemini 2.5 Flash takes over. Using Automatic Function Calling (AFC), Gemini orchestrates a multi-step agentic loop:

  1. It calls get_scene_comparison to pull the raw vector metrics from ClickHouse.
  2. It executes spatial grounding to locate the physical bounding box [ymin, xmin, ymax, xmax] of the defect.
  3. It calls estimate_production_impact to execute the financial verdict:

$$\Delta_{NetCapital} = Cost_{Reshoot} - Cost_{VFXFix}$$

If the digital VFX repair ($1,005) is significantly cheaper than a physical reshoot ($35,000), Gemini emits the ultimate directive on the console: 🟢 WRAP SET AUTHORIZED, saving the studio massive amounts of capital.

🚧 Challenges we ran into

1. The Cartesian Explosion in Vector Telemetry: When ingesting multiple takes from different cameras into a ClickHouse MergeTree, asynchronous compaction caused duplicate frame indices during the ingestion phase. When executing the JOINs to compare Take 01 vs Take 02, this triggered massive Cartesian explosions, multiplying rows and crashing our analytical queries. We engineered a deterministic SQL solution using LIMIT 1 BY frame_idx subqueries to guarantee a strict 1:1 frame comparison at lightning speed.

2. Acoustic Synchronization: Aligning 48,000 Hz audio samples precisely with 24 FPS SMPTE timecodes proved incredibly difficult. If the audio drifted even slightly, the room tone delta would trigger false positives. We solved this by building a custom NumPy extraction pipeline that segments the audio array precisely to match the visual frame windows, ensuring zero drift.

3. Taming Hallucinations in Financial Logic: We could not afford an LLM hallucinating production advice. We solved this by strictly limiting Gemini's context window to the deterministic SQL outputs provided by ClickHouse. Gemini is not guessing the error; it is simply interpreting the rigid mathematical deltas calculated by the database.

🏆 Accomplishments that we're proud of

  • 100% Zero Third-Party AI APIs: We relied exclusively on Google Cloud (Vertex AI) and ClickHouse Cloud. No external wrappers or third-party cognitive services.
  • The "Wow" Factor UI: Building a front-end Single Page Application (SPA) that legitimately looks and feels like a $100M SaaS product. We meticulously implemented DaVinci Resolve aesthetics, complete with a live interactive <canvas> for difference mattes and neon visual overlays.
  • Real Industry Outputs: We don't just output text on a screen. Our backend dynamically generates mathematically accurate ASC-CDL XML files, bridging the gap between set data and post-production software.

📚 What we learned

We learned the sheer, terrifying speed of ClickHouse Cloud. Transitioning from traditional relational databases (like PostgreSQL) to a columnar OLAP database for high-frequency video telemetry was a massive paradigm shift for us.

We also discovered how incredibly powerful Google's Automatic Function Calling is when you need an LLM to orchestrate a complex pipeline rather than just acting as a conversational chatbot. Watching Gemini query ClickHouse autonomously to decide if a $35,000 stage hold was necessary felt like interacting with the future of filmmaking.

🔮 What's next for CineAudit Sentinel

This hackathon is just our pilot episode. The next phase for CineAudit Sentinel involves:

  1. Live SDI Camera Ingestion: Bypassing MP4 uploads entirely by connecting the engine directly to the live SDI feed of ARRI and RED digital cinema cameras on set.
  2. Post-Production API Integration: Pushing the ASC-CDL color grades, bounding boxes, and timeline markers directly into the editor's DaVinci Resolve suite over the local network via Python API.
  3. Multi-Camera Sync: Expanding the ClickHouse telemetry schema to audit Continuity across 4 or more simultaneous camera angles using multi-vector clustering.

The future of Agentic AI isn't just about making workflows faster; it's about protecting enterprise capital. And on a Hollywood film set, CineAudit Sentinel is the ultimate guardian.

Built With

Share this project:

Updates

Submission history