SceneDNA

SceneDNA is pre-release audience intelligence for studios. Gemini extracts a scene's creative DNA, ClickHouse finds which patterns historically lose viewers, and editors get fixes before release.

Comment

Inspiration

Studios learn that a scene lost its audience after release, when retention dashboards light up and nothing can be changed. Editors already have instincts about which scenes drag, but no evidence to back them before a cut ships. We wanted a tool that turns historical viewer behavior into a pre-release risk check, and does it across many scenes so it finds recurring patterns rather than explaining one bad minute.

What it does

SceneDNA is pre-release audience intelligence for editors.

  1. Scene Analyst. Gemini watches an uploaded scene cut and extracts its creative DNA: dialogue ratio, pacing in words per minute, motion intensity, action level, camera motion, music intensity, emotional intensity, average shot length, visible characters, and a plain summary.
  2. Pattern Analyst. The agent queries 4.8 million historical viewer retention events in ClickHouse Cloud through the official ClickHouse MCP server. It computes drop-off rate and hazard ratio for every age group and device on scenes sharing the cut's profile, a drop-off curve across the runtime, and cross-scene pattern discovery. Gemini is also given a clickhouse_run_query tool and writes its own verification queries.
  3. Risk Analyst. The new scene is compared against the discovered risk factors and gets a HIGH, MEDIUM, or LOW verdict with the match count and the historical exit rate of the affected cohort.
  4. Strategist. Gemini synthesizes a structured prescription: exact cut ranges, camera direction, and the estimated exit rate after the edit, phrased as correlation, never causation.

The studio UI shows an ask-anything bar that answers questions by querying the warehouse, the scene feature card, historical pattern discovery, the risk scan, the recommendations, and a ClickHouse evidence panel with the exact SQL, rows scanned, latency, and the full agent tool trace. A Live / Demo badge makes it explicit when real services are in use.

Example finding: dialogue-heavy scenes over four minutes with a static camera are associated with a 23.5 percent exit rate among 18 to 24 year old mobile viewers, about 2.8 times the global baseline.

How we built it

  • Gemini via google-genai, served through Vertex AI on Google Cloud. Multimodal video understanding with a typed response schema, structured tool calling into ClickHouse, and a bounded reasoning loop that evaluates query results and decides whether to query again. A model fallback chain and per-call output caps keep it resilient.
  • ClickHouse Cloud through mcp-clickhouse. The backend keeps one MCP server process alive over stdio and every runtime query is an MCP run_query tool call. Schema discovery uses list_tables. Each call is recorded with latency and row counts.
  • FastAPI orchestrates the pipeline and serves the compiled React + TypeScript + Tailwind studio UI from one container on Cloud Run.
  • Data. A vectorized NumPy seeder generates 4.8 million synthetic viewer events with deliberately planted, discoverable patterns, plus a scene library with creative attributes. No real audience data is used; uploaded clips are analyzed at request time and never stored.

Challenges we ran into

  • Gemini model availability fluctuated during the build. A fallback chain across models, a short result cache, and honest fallback labeling kept the app usable while the live path stayed primary.
  • Very short clips matched no library scene in the strict profile band, so the matcher widens to the nearest creative profiles and says so in the UI.
  • Routing everything through MCP meant rendering parameterized SQL client-side, since the run_query tool takes a single string. The UI shows both the parameterized template and the bound query.

Accomplishments

Every number on screen comes from a real query at runtime. Gemini writes and executes its own ClickHouse queries through MCP. The whole workflow, from video to verdict, completes in under a minute.

What we learned

Structured output changes everything for a studio tool. Typed cut ranges render directly on the timeline and in the recommendations, where free text would have been a wall of prose. And warm MCP sessions are fast: 100 to 400 ms per aggregation over 4.8 million rows, which is quick enough to run five queries per analysis and still feel live.

What's next

A larger scene library across more titles, per-episode pattern drift, and a review workflow so editors can accept or reject each recommended cut and feed the outcome back into the warehouse.

Built With

Share this project:

Updates