CineAnalytics AI: Real-Time Telemetry & AI Studio Director

CineAnalytics AI: Autonomous AI Studio Director powered by Gemini 2.5 & ClickHouse. It unifies real-time box office, video streaming QoE, & social sentiment into autonomous campaign decisions.

Comment

🎬 CineAnalytics AI: Real-Time Telemetry & Autonomous AI Studio Director

💡 Inspiration

In the modern entertainment landscape, releasing a cinematic blockbuster is a high-stakes, multi-million-dollar operation. Yet major studios and streaming networks still manage distribution using fragmented, backward-looking reports:

  • Theatrical Box Office: Ticket sales and theater occupancy rates are collected hours or days after showtimes.
  • Streaming Video Quality of Experience (QoE): HLS and DASH telemetry (bitrate drops, buffering ratios, player stalls) are locked inside CDN metrics dashboards.
  • Audience Social Sentiment: Viral reactions, trailer feedback, and audience memes on TikTok, YouTube, and X evolve in minutes, but marketing budgets remain static.

When a movie opens, studios lose millions because these three pillars live in isolated silos. If a streaming premiere suffers playback buffering during a pivotal action scene, or if a specific 10-second trailer clip goes viral on TikTok, manual response times are far too slow.

We were inspired by the "Lights. Camera. Code." challenge to step onto the virtual studio backlot and build CineAnalytics AI—an autonomous AI Studio Director powered by Gemini 2.5 Flash and ClickHouse that unifies real-time media telemetry and takes deterministic, closed-loop actions to optimize box office yield and streaming retention.

🎯 What It Does

CineAnalytics AI acts as an autonomous command center and co-director for studio executives:

  1. Unifies Real-Time Telemetry: Ingests theatrical ticket transactions, scene-by-scene video streaming telemetry, and audience sentiment into ClickHouse Medallion architecture tables (box_office_mart, streaming_qos_mart, sentiment_mart).
  2. Autonomous Multi-Agent Fleet: Powered by Gemini 2.5 Flash and Google Agent Development Kit (ADK) patterns, a hierarchical agent crew collaborates in real time:
    • RootDirectorAgent: Master orchestrator managing executive analysis and report generation.
    • BoxOfficeAnalystAgent: Calculates ticket velocity, regional occupancy rates, and opening weekend projections.
    • StreamingQoSAgent: Analyzes streaming health and identifies scene-by-scene retention bottlenecks.
    • SentimentAnalystAgent: Tracks viral scene hashtags and sentiment trends across platforms.
    • MarketingOrchestratorAgent: Formulates and executes automated budget shifts and distribution tweaks.
  3. Closed-Loop Action Execution: When the system detects a drop in stream retention or a viral social surge, Gemini autonomously executes campaign actions—such as shifting advertising capital toward viral clips or reallocating late-night IMAX showtimes—and records every action in an immutable ClickHouse audit log (campaign_decisions_log).
  4. Interactive Studio Command Center: A sleek, dark-mode glassmorphism dashboard providing live KPI tracking, real-time agent trajectory inspection, a ClickHouse SQL sandbox, and a live scenario simulation engine (TikTok Viral Surge, Edge CDN Congestion, IMAX Sellout).

📐 Mathematical Formulation & Telemetry Models

To quantify viewer retention and marketing efficiency deterministically, CineAnalytics AI evaluates streaming health and budget allocations using real-time formulas:

1. Streaming Quality of Experience (QoE) Index

Viewer retention across individual scenes is modeled as a function of average bitrate $B_s$, buffering duration $T_{buf}$, and total playback time $T_{play}$:

$$ QoE_s = 100 \cdot \left( \frac{\min(B_s, B_{target})}{B_{target}} \right) \cdot \exp\left(-\lambda \cdot \frac{T_{buf}}{T_{play}}\right) $$

  • $\lambda$ is the viewer impatience penalty factor ($\lambda \approx 4.5$).
  • When $QoE_s < 75$, the agent autonomously triggers edge CDN cache routing to mitigate churn.

2. Viral Sentiment Velocity & Campaign Reallocation

The autonomous marketing budget shift $\Delta M$ toward a viral scene hook is computed using the sentiment score $S \in [0, 1]$ and mention velocity $V = \frac{d(\text{mentions})}{dt}$:

$$ \Delta M = M_{base} \cdot \left(1 + \alpha \cdot S \cdot \log_{10}(1 + V)\right) $$

  • $\alpha$ is the studio allocation multiplier, ensuring budget scaling remains bounded and ROI-positive.

🛠️ How We Built It

  • Google Cloud & Gemini:
    • Leveraged Gemini 2.5 Flash for low-latency, multi-step function calling loops.
    • Designed structured agent tools conforming to Google Agent Development Kit (ADK) conventions (get_box_office_analytics, get_streaming_qos_telemetry, get_social_sentiment_analytics, execute_clickhouse_sql, execute_studio_campaign_decision).
  • ClickHouse Real-Time Analytics Engine:
    • Designed a Medallion Architecture (Raw Ingestion $\rightarrow$ Aggregated Marts) using MergeTree engines.
    • Integrated via clickhouse-connect and Model Context Protocol (MCP) tool patterns.
    • Built an embedded high-throughput analytical fallback engine ensuring instant local reproducibility.
  • Full-Stack Application:
    • Backend: FastAPI REST service with asynchronous event streaming and simulation endpoints.
    • Frontend: React and Vite with Lucide icons, glassmorphism CSS design system, live telemetry tickers, and an interactive ClickHouse SQL terminal.
  • Evaluation & Testing:
    • Programmatic evaluation suite (evals/eval_suite.py) validating tool trajectory accuracy, schema integrity, and SQL safety guardrails.

🧗 Challenges We Faced

  1. Heterogeneous Telemetry Velocities: Streaming QoS events arrive in millisecond intervals, while box office tickets transact in bursts and social media metrics fluctuate continuously. Structuring ClickHouse analytical marts to serve pre-aggregated, low-latency views allowed our agents to reason over high-frequency data without token exhaustion.
  2. Ensuring Agent Determinism & Safety: In enterprise studio environments, autonomous agents cannot run destructive database operations or hallucinate financial actions. We implemented strict read-only SQL validation (SELECT-only enforcement) and bounded budget guardrails for every campaign decision.
  3. Cross-Platform Reproducibility: Creating a seamless developer experience that runs either against a live ClickHouse Cloud cluster or entirely self-contained in local development required crafting an intelligent dual-mode database client.

📚 What We Learned

  • Agent-Facing Analytics: Providing LLMs with columnar pre-aggregated data marts rather than raw operational tables dramatically increases tool-calling reliability and reduces reasoning latency.
  • The Synergy of Gemini & ClickHouse: Gemini 2.5 Flash excels at synthesizing complex, multi-variable analytical results into concise, executive-ready action plans when grounded with deterministic columnar queries.
  • Autonomous Studio Governance: Media workflows benefit enormously from human-in-the-loop oversight combined with automated baseline execution—giving directors transparency into why a decision was made via step-by-step agent trajectories.

🚀 What's Next for CineAnalytics AI

  • Unreal Engine & Virtual Production Sidecars: Ingesting real-time GPU render telemetry from virtual production stages directly into ClickHouse.
  • Agent-to-Agent (A2A) Protocols: Connecting the studio director agent with external ad-exchange agents to programmatically place and bid on media campaigns in real time.
  • Predictive Box Office Forecasting: Training time-series forecasting models within ClickHouse using historical theatrical comps to predict lifetime gross from opening night velocity.

Built With

Share this project:

Updates

Submission history