CinePulse: Real-Time OTT Intelligence Agent

Inspiration

Streaming platforms lose millions of dollars and countless subscribers every year due to a critical blind spot: platforms track when viewers leave, but creative teams rarely understand why. Traditional relational databases buckle under the weight of high-frequency viewer telemetry, while editorial decisions remain completely detached from real-time viewing behavior. Creative teams receive retention post-mortems weeks or months after launch far too late to save an underperforming title.We built CinePulse to bridge high-volume viewer behavior with creative media decisions. By combining vectorized columnar streaming analytics with autonomous Gemini AI agents and automated FFmpeg editing pipelines, CinePulse transforms passive telemetry into immediate creative action.

What it does

CinePulse is an autonomous real-time streaming intelligence agent designed for modern OTT platforms:Sub-10ms Streaming Telemetry: Ingests high-frequency viewer interactions (play, pause, seek, scene_enter, scene_exit, exit, completion) from thousands of concurrent sessions directly into ClickHouse Cloud.Autonomous Gemini AI Agent: When a retention drop is flagged (e.g., Scene 17 drop-off), a Google Gemini 1.5 agent executes 8 native backend tools to inspect telemetry, analyze dialogue density, evaluate pacing, and diagnose the root cause with zero hallucination.Actionable Creative Directives: Delivers evidence-grounded recommendations for film editors (trimming exposition, adjusting tempo) and marketing teams (audience re-targeting hooks). Automated FFmpeg Trailer Generation: Identifies peak retention scenes and programmatically cuts and stitches an optimized teaser trailer in seconds. Cinematic Live Dashboard: Built with Next.js 15, providing real-time WebSocket telemetry, dynamic retention graphs, live viewer counters, and an interactive agent investigation studio.

How we built it

Dual-Database Architecture:ClickHouse Cloud: Serves as the vectorized columnar event store, aggregating hundreds of thousands of viewer events in under 10 ms using vectorized SQL queries.SQLite: Acts as an ultra-fast, in-process transactional store for application metadata, scene dialogue densities, pacing scores, and trailer rendering jobs.ntelligence Layer: Google Gemini 1.5 integrated via native function calling across 8 specialized backend tools (query_viewership, get_scene_metrics, detect_dropoffs, compare_scenes, get_content_metadata, get_top_performing_scenes,generate_trailer_strategy, generate_marketing_strategy).Media Automation: FFmpeg running via safe backend subprocesses to clip, concatenate, and export MP4 teaser cuts directly to the browser.Backend: FastAPI with asynchronous WebSockets dispatching real-time event telemetry. Frontend: Next.js 15 App Router, Tailwind CSS, and Lucide icons in a dark cinematic interface.

Challenges we ran into

High-Concurrency Analytical Latency: Standard transactional databases locked up when simultaneously ingesting raw viewer clickstreams and computing real-time survival curves. We solved this by routing all event ingestion to ClickHouse MergeTree tables, keeping query latencies under 10 milliseconds.Grounding Agent Reasoning: Early LLM tests produced generic creative feedback without statistical backing. We resolved this by giving the Gemini agent strict native backend tools, forcing it to fetch empirical SQL data and script metrics before issuing recommendations.Zero-GPU Lightweight Media Processing: Video rendering pipelines often demand heavy cloud GPUs. We optimized our FFmpeg pipeline with safe subprocess timeouts and stream copying to run cleanly on standard laptops with 8 GB RAM.

Accomplishments that we're proud of

Engineered a complete closed-loop workflow: Viewer Telemetry Anomaly Detection , Gemini Investigation , Automated FFmpeg Trailer Rendering.Maintained sub-10ms aggregation speed over 50,000+ simulated viewer events.Implemented a zero-crash local analytical fallback mode that enables seamless offline judging without external cloud dependencies.

What we learned

Columnar architectures like ClickHouse drastically outperform relational engines for cohort survival and drop-off analytics.Grounded agentic workflows with native function calling produce significantly more actionable, trustworthy insights than raw prompt engineering.Media optimization pipelines can be executed efficiently on consumer hardware when paired with targeted telemetry selection.

What's next for CinePulse: Real-Time OTT Intelligence Agent

Multimodal Computer Vision Analysis: Integrating vision models to evaluate visual pacing, lighting, and camera movement alongside script dialogue density.Dynamic A/B Scene Delivery: Serving alternative scene edits dynamically to segmented viewer cohorts to test retention fixes before finalizing master cuts.Direct Studio CMS Connectors: Building one-click export integrations for YouTube Studio, Vimeo OTT, and enterprise media asset management systems.