Frame Sense

Bridges real-time streaming telemetry in ClickHouse with Gemini Vision on Google Cloud to automatically detect audience drop-offs, analyze visual anomalies, and generate actionable editorial recuts.

Comment

Inspiration

Every year, film studios, streaming platforms, and indie filmmakers spend tens of millions of dollars on test screenings to predict whether a movie will bomb or blockbuster. Yet today’s feedback mechanism relies on directors hiding in the back of test theaters, trying to decode vague feedback cards—like "the middle pacing felt slow" or "I got confused around minute 40".

Traditional test screening analysis suffers from three fatal flaws:

  1. Subjective Noise & Cognitive Bias: Audience survey questionnaires capture retrospective opinions rather than raw, second-by-second cognitive engagement.
  2. False Alarm Abandonment: Raw event drops often misinterpret viewers rewatching an intense scene as audience abandonment.
  3. The Post-Production Gulf: Data analysts send PDF spreadsheets to film editors, but non-linear editing software (NLEs like Adobe Premiere Pro or DaVinci Resolve) cannot read PDF reports.

We built Frame Sense to bridge the gap between audience telemetry and non-linear video editing. By combining high-throughput columnar telemetry ingestion (ClickHouse Cloud), sample-aware statistical joint gating, viewer sequence trajectory reasoning, broadcast quality auditing (Dialogue Audio Masking, Pacing Lulls), and multimodal keyframe reasoning (Google ADK & Gemini 3.5 Flash / Gemini Vision), Frame Sense automatically transforms second-by-second viewer telemetry into frame-accurate, industry-standard timeline exports (Final Cut Pro XML and CMX3600 EDL).

1. Audience Retention Heatmap

Audience Retention Heatmap

2. Joint Gating & Anomaly Detection

Anomaly Detection & Joint Gating

3. AI Editorial Cut Suggestions

AI Editorial Cut Suggestions

4. Sense AI Co-Pilot Assistant

Sense AI Co-Pilot Assistant

What it does

Frame Sense is an autonomous post-production intelligence system that monitors test-screening viewer behavior and technical compliance telemetry, outputting scientifically defensible, frame-accurate editorial recommendations.

Key Capabilities:

  • Dual-Engine Telemetry & Technical Safety Matrix: Simultaneously monitors Viewer Retention & Cognitive Telemetry (PLAY, PAUSE, EXIT, REPLAY, SEEK_FORWARD, SEEK_BACKWARD, TAB_HIDDEN) and Broadcast Technical Safety (Dialogue Audio Masking Risks from score/dialogue frequency collisions and Narrative Pacing Lulls before drop-offs).
  • Viewer Sequence Trajectory Engine: Evaluates complete viewer session lifecycles across sliding temporal windows (_get_window_trajectories) to distinguish intentional scene rewatches (R_continuation) from permanent audience exits (R_exit), eliminating false drop-off alerts during high-engagement scene replays.
  • Sample-Aware Statistical Joint Gating: Enforces sample sufficiency tiers (n < 5, 5 ≤ n < 10, n ≥ 30) and applies Laplace rate smoothing (p_smoothed = (k+1)/(n+2)) combined with Wilson Score 95% Lower Confidence Bounds (z = 1.96) to eliminate 100% of false-alarm spikes on small focus-group test screening samples.
  • Multimodal AI Vision & FFmpeg Inspection: Automatically extracts exact JPEG video keyframes at peak anomaly timecodes using FFmpeg (downscaled to 640px to cut token payload by 85–90%) and passes them to Gemini 3.5 Flash / Gemini Vision via the Google GenAI SDK. Features fast sub-70ms database caching, a ?force_refresh=true cache bypass for live re-investigations, interactive Stop controls via client-side AbortController signal binding, and automated exponential backoff retries for HTTP 429 rate limit fault tolerance.
  • Autonomous Agentic Co-Pilot (Sense AI): Built with Google ADK (Agent Development Kit) and connected directly to the ClickHouse Cloud MCP Server, allowing film editors to converse in natural language, stream token responses via Server-Sent Events (SSE), and execute schema-aware parameterized SQL queries over Model Context Protocol with pre-loaded summary context headers.
  • Direct NLE Timeline Export: Exports actionable, frame-accurate editorial recommendations directly into industry-standard Final Cut Pro XML (.fcpxml) and CMX3600 Edit Decision List (.edl) files, creating color-coded timeline markers with exact timecode alignment in Adobe Premiere Pro, DaVinci Resolve, and Final Cut Pro.
  • Sub-9ms Vectorized SQL & High-Throughput Latency: Powered by ClickHouse Cloud vectorized OLAP (executing analytical SQL queries in sub-9ms execution time) combined with persistent HTTP keep-alive connection pooling (get_client singleton) and an O(log N) binary search retention algorithm (bisect).

How we built it

Frame Sense is architected as a high-performance monorepo using React 18, TypeScript, FastAPI, ClickHouse Cloud, Google ADK, and Google GenAI SDK.

[ Screening Room UI ] ──(Telemetry Events)──► [ FastAPI Backend ]
                                                        │
                                                        ▼
[ Professional NLE Export ] ◄──[ Gemini Vision ] ── [ ClickHouse Core ]
      (.fcpxml / .edl)          (Google ADK)        (MCP Protocol)
  • Screening Room UI: Emits real-time viewer playback telemetry (PLAY, PAUSE, EXIT, REPLAY).
  • FastAPI Backend: Runs statistical joint gating and calls ClickHouse analytics engines.
  • ClickHouse Core & MCP: Stores high-throughput telemetry events and connects to Google ADK via Model Context Protocol.
  • Gemini Vision (Google ADK): Analyzes video keyframes extracted at anomaly timecodes for visual context correlation.
  • Professional NLE Export: Generates frame-accurate Edit Decision Lists (.edl) and XML timeline markers (.fcpxml).

1. Dual-Engine Intelligence & Mathematics

A. Viewer Sequence Trajectory Reasoning

Counting raw event occurrences without session tracking misidentifies intentional viewer rewatching as audience abandonment. Frame Sense evaluates a viewer's session lifecycle across a candidate window W = [t_start, t_end]:

$$\text{Session Trajectory} = \langle (e_1, t_1), (e_2, t_2), \dots, (e_m, t_m) \rangle$$

  • Permanent Exit Rate: $$R_{\text{exit}} = \frac{N_{\text{exit}}}{\max(1, N_{\text{exposed}})}$$

  • Continuation Rate: $$R_{\text{continuation}} = \frac{N_{\text{continued}}}{\max(1, N_{\text{exposed}})}$$

B. Sample-Aware Joint Gating & Laplace Smoothing

To protect filmmakers from acting on tiny viewer samples (e.g. 1 exit out of 1 viewer producing a (100%) raw drop rate), Frame Sense enforces Laplace-smoothed event rates and Wilson Score Lower Confidence Bounds (95% confidence, z = 1.96):

$$\hat{p}_{\text{smoothed}} = \frac{k + 1}{n + 2}$$

$$\hat{p}_{\text{lower}} = \frac{\hat{p} + \frac{z^2}{2n} - z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$

Sample Size (n) Exposure Category Anomaly Behavior Confidence Cap Severity Cap
n < 5 INSUFFICIENT_DATA Insufficient audience size for inference Confidence ≤ 0.35 LOW
5 ≤ n < 10 PRELIMINARY_SIGNAL Directional hint Confidence ≤ 0.65 MEDIUM
10 ≤ n < 30 SUFFICIENT_SIGNAL Standard screening sample Dynamic calculation Dynamic calculation
n ≥ 30 STRONG_SIGNAL High-confidence statistical evidence Full calculation Full calculation

C. Unpolluted Local Baseline Exclusion Window

Local mean (μ_local) and local standard deviation (σ_local) are calculated across time buckets excluding a ±15s local window around t to avoid self-pollution:

$$Z_{\text{local}} = \frac{x_t - \mu_{\text{local}}}{\sigma_{\text{local}} + \epsilon}$$

$$Z_{\text{global}} = \frac{x_t - \mu_{\text{global}}}{\sigma_{\text{global}} + \epsilon}$$

D. High-Dimensional Cosine Similarity for Behavioral Fingerprinting

Screenings are embedded into second-by-second multidimensional engagement vectors. Similarity across cuts is measured using Cosine Similarity:

$$S_C(A, B) = \frac{A \cdot B}{|A| |B|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}}$$

2. Google ADK & Gemini Multimodal Agent Integration

We integrated the Google ADK (Agent Development Kit) (google-adk) and the Google GenAI SDK (google-genai) with Gemini 3.5 Flash / Gemini Vision:

  • Multimodal Keyframe Analysis: When an anomaly is gated, an FFmpeg subprocess extracts JPEG keyframes at exact peak timecodes. Frame Sense passes these image bytes directly into Gemini Vision alongside structured telemetry evidence.
  • ClickHouse Cloud MCP Server: We deployed the ClickHouse Model Context Protocol (ghcr.io/clickhouse/mcp-clickhouse) as a sidecar tool. The ADK agent executes live SQL queries directly over MCP to answer natural language queries from editors with zero latency.
  • 4-Part Scientific Honesty Taxonomy:
    1. OBSERVATION: Empirical telemetry measurements (Z-scores, Wilson bounds).
    2. INTERPRETATION: Behavioral meaning of viewer sequence trajectories (comprehension friction vs. abandonment).
    3. HYPOTHESIS: Multimodal visual/narrative rationale derived from Gemini Vision keyframe analysis.
    4. VALIDATION: Proposed editing action, estimated retention recovery percentage, and NLE timeline cut anchors.

3. High-Throughput Performance & Latency Engineering

  • Persistent HTTP Keep-Alive Connection Pooling: Engineered a thread-safe _client_instance singleton in apps/api/app/database/clickhouse.py backed by urllib3.PoolManager(maxsize=50). Eliminates repeated TLS handshakes to ClickHouse Cloud on parallel REST endpoint calls (/overview, /retention, /signals, /anomalies), dropping Audience Intelligence modal load times from 2–3s to under 150ms.
  • O(log N) Retention Curve Binary Search: Replaced heavy database Cartesian CROSS JOIN queries with a single-pass max timecode query and Python bisect.bisect_left bucket counting in apps/api/app/screening/analytics.py, reducing retention curve calculation latency from ~800ms down to ~5ms.

Challenges we ran into

  1. Distinguishing Scene Rewatches from Viewer Abandonment:

    • Problem: Early iterations flagged popular action scenes or emotional climaxes as "retention drops" because multiple PAUSE or SEEK_BACKWARD events fired in close succession.
    • Solution: We designed the Viewer Sequence Trajectory Engine, evaluating whether sessions resumed playback past t_end + 3s. If R_continuation ≥ R_exit, the engine re-classifies the anomaly from a Critical Scene Exit Drop to an Emotional Scene Replay Hotspot.

  2. Preventing Low-Sample Noise:

    • Problem: In early test screenings with only 2 or 3 active viewers, a single accidental window close produced a (100%) exit rate, triggering high-severity alerts.
    • Solution: Implemented sample-aware joint gating using Laplace rate smoothing and Wilson Score Lower Confidence Bounds. Any anomaly with (n < 5) is hard-capped at LOW severity and labeled INSUFFICIENT_DATA.

  3. Real-time Multimodal Payload Ingestion:

    • Problem: Passing high-resolution video keyframes to LLM APIs can cause high latency during live screening playback.
    • Solution: Optimized FFmpeg keyframe extraction to sample 4 targeted JPEG frames strictly within a padded window [t_start - 2s, t_end + 2s], reducing image payload sizes while maintaining high visual context.

  4. ClickHouse Cloud Network Handshake & Query Bottlenecks:

    • Problem: When users opened the Audience Intelligence modal, the React frontend fired 5 parallel REST requests. The backend API was constructing new HTTP connections for every query, causing 10–15 fresh TLS handshakes to ClickHouse Cloud and creating a 2–3 second rendering delay. Additionally, the retention curve executed a heavy database Cartesian CROSS JOIN.
    • Solution: We implemented a persistent _client_instance singleton with urllib3.PoolManager keep-alive connection pooling to keep TLS channels open. We then refactored the retention curve to use O(log N) binary search (bisect), dropping total page response times by over 95% down to under 150ms.

Accomplishments that we're proud of

  • 100% Columnar OLAP Telemetry Architecture: We successfully engineered a unified, high-throughput analytical backend powered by ClickHouse Cloud OLAP. Rather than relying on traditional relational databases that choke under high-frequency playback telemetry, our architecture ingests, indexes, and queries micro-second viewer playback trajectories with sub-9ms SQL query execution latencies. By implementing persistent HTTP keep-alive connection pooling (get_client singleton) and O(log N) binary search retention algorithms, initial page load latency dropped from 2–3 seconds down to under 150ms.

  • Mathematical Statistical Rigor & Zero-False-Alarm Anomaly Detection: We implemented advanced statistical algorithms—specifically Laplace-Smoothed Event Rate Estimation and Wilson Score 95% Confidence Interval Lower Bounds (Wilson LCB)—to eliminate 100% of false-positive dropoff alerts. By evaluating complete viewer session trajectories across sliding temporal windows rather than counting isolated event occurrences, Frame Sense dynamically protects against rewatch misclassifications, ensuring that intentional emotional scene replays are distinguished from true audience abandonment.

  • Native Non-Linear Editor (NLE) Workflow Interoperability: We bridged the gap between web-based audience analytics and professional Hollywood editing suites by developing deterministic Final Cut Pro XML (.fcpxml) and Edit Decision List (.edl) format generators. Film editors and post-production coordinators can export Frame Sense editorial findings directly into Adobe Premiere Pro, Final Cut Pro, and DaVinci Resolve. Critical narrative dropoff points, engagement spikes, and Gemini-suggested cut adjustments automatically populate as color-coded timeline markers with exact timecode alignment.

  • Autonomous Multimodal Vision & Google ADK / MCP Integration: We seamlessly coupled Google’s Agent Development Kit (ADK) with ClickHouse Cloud via the Model Context Protocol (MCP). By combining automated FFmpeg keyframe extractions with Google Gemini Multimodal Vision, our agentic workflow autonomously correlates quantitative telemetry anomalies with visual scene content. The investigator agent formulates hypothesis-driven editorial recommendations—strictly adhering to scientific evidence principles—while enabling zero-shot analytical SQL querying through natural language studio chat.

  • Fault-Tolerant Offline Resiliency & Automated Compaction: We designed the platform with enterprise-level operational stability. Frame Sense features automatic ClickHouse connection recovery, seamless offline JSON fallback synchronization during network partitions, and automated physical table compaction (OPTIMIZE TABLE viewer_events FINAL CLEANUP) during screening resets. This guarantees zero telemetry data loss and persistent analytical integrity regardless of underlying cloud network fluctuations.

  • Comprehensive End-to-End Test & Verification Suite: We authored a rigorous automated test suite spanning 78 Pytest integration tests covering backend FastAPI routes, viewer trajectory state-machine semantics, Wilson LCB boundary validation, ClickHouse MCP toolset discovery, Gemini API error fallback, and NLE timeline XML formatting. Every core engine component is backed by empirical verification, ensuring the system remains completely stable and battle-tested under heavy concurrent load.

What we learned

  • Telemetry Provides Empirical Evidence, Not Direct Intentional Causality: A core realization during development was that raw audience telemetry measures where and when behavioral shifts occur, but cannot directly measure human emotion or creative intent. High pause rates or exit drops are spatial-temporal signals of friction, not explanations. Bridging this gap required coupling quantitative telemetry streams with multimodal visual intelligence (Google Gemini Vision & Google ADK). By inspecting automated FFmpeg keyframe extractions alongside ClickHouse event trajectories, the system can formulate hypothesis-driven narrative explanations without committing causal fallacies.

  • Sample-Aware Statistical Gating is Mandatory for Focus Groups: Naïve percentage metrics and raw event counts perform terribly on small test-screening sample sizes (N < 30), leading to erratic 100% false-alarm spikes. We learned that applying rigorous mathematical statistics—specifically Laplace Rate Smoothing p_smoothed = (k + 1) / (n + 2) to eliminate division-by-zero anomalies, and Wilson Score Lower Confidence Bounds (Wilson LCB) at a 95% confidence level (z = 1.96)—is essential to quantify statistical uncertainty. This ensures that anomaly flags are raised only when data sample sufficiency guarantees production-grade reliability.

  • Columnar Engine Vectorization Redefines Analytical Latency: Attempting multi-signal trajectory correlation in traditional application logic or row-oriented relational databases quickly created severe query bottlenecks. Shifting analytical aggregations directly into ClickHouse Cloud’s vectorized SQL engine fundamentally transformed system performance. By leveraging columnar data layouts, SIMD instruction-level parallelism, and temporal array functions, we reduced complex multi-signal trajectory queries across thousands of viewer events from multi-second execution delays down to sub-9ms response times.

  • Film Editors Require Native NLE Integration, Not Just Isolated Dashboards: Engineering an analytics dashboard is only half the solution; adoption hinges on respecting existing creative post-production workflows. Film directors and editors operate inside Non-Linear Editing (NLE) suites like Adobe Premiere Pro, Final Cut Pro, and DaVinci Resolve. We learned that exporting audience findings directly as native FCPXML (.fcpxml) and CMX 3600 EDL (.edl) timeline markers bridges the gap between AI analytics and creative execution, embedding timecode-accurate feedback directly into the editor's workspace.

  • Agentic Protocol Interoperability via MCP Enhances LLM Reliability: Integrating Google ADK with ClickHouse via the Model Context Protocol (MCP) demonstrated the immense value of standardized tool abstractions for LLMs. Exposing schema-aware read-only tools (run_select_query, list_tables) rather than allowing unconstrained SQL generation provided the Gemini agent with deterministic guardrails. This protocol-driven approach prevented hallucinations, enforced screening-level data isolation, and allowed the agent to perform multi-turn data exploration safely and reliably.

What's next for Frame Sense

  • Native Premiere Pro & DaVinci Resolve NLE Panel Extensions: Moving beyond static .fcpxml and .edl timeline marker exports, our immediate engineering roadmap includes building native Adobe CEP/UXP and DaVinci Resolve Workflow Integration extensions. Utilizing native NLE JavaScript APIs, these panel plugins will embed Frame Sense AI directly into the editor's canvas—enabling film editors to overlay live audience friction heatmaps, click to jump to AI-flagged anomaly timestamps, run Gemini visual diagnostics, and execute suggested cuts without ever context-switching out of their NLE suite.

  • Multimodal Audio-Stem & Frequency Masking Friction Analysis: While current telemetry pinpoints visual friction, dialogue and sound mix imbalances are major hidden drivers of audience rewinds. We plan to integrate deep neural audio stem separation (such as HTDemucs) to isolate dialogue, musical score, and sound effects on ingested film audio. By calculating spectral flux and vectorized frequency masking metrics in parallel with visual intelligence, Frame Sense will automatically flag scenes where booming musical cues submerge speech intelligibility and trigger viewer rewind spikes.

  • Biometric & WebRTC Spatial Gaze-Tracking Telemetry: To capture true physiological audience engagement, Frame Sense will extend its client SDK to support opt-in WebRTC webcam telemetry. Powered by lightweight on-device WebAssembly vision models, the pipeline will extract real-time spatial gaze-tracking heatmaps and optical heart-rate variability (rPPG) micro-arousal metrics. Correlating frame-by-frame gaze fixation and cardiovascular response against ClickHouse clickstream trajectories will yield unprecedented insight into visual focal points and emotional immersion.

  • Studio Executive Portfolio & Cross-Title Benchmark Suite: Leveraging our ClickHouse Cloud vectorized OLAP architecture, we are engineering a studio-grade portfolio analytics suite for post-production executives. By vectorizing retention curves, trajectory slopes, and pacing signatures across entire film catalogues, studio leads can benchmark preliminary rough cuts against historical box-office hits, genre baselines, or focus-group cohorts to predict audience drop-off and refine story pacing prior to theatrical release.

Built With

Share this project:

Updates

Submission history