CineAI

Opentelemetry Log
Opentelemetry Log
UI Dashboard
UI Dashboard
UI Dashboard
UI Dashboard
Execution in Google ADK web

💡 Inspiration

On a film set, time isn't just money, it's thousands of dollars per minute. When an unexpected monsoon storm hits, a primary camera sensor recalibration is delayed, or a municipal permit gets stalled, everything halts. The Assistant Director (AD) is suddenly under extreme pressure to juggle call sheets, cross-reference actor availabilities, check alternate set permits, verify equipment schedules, and rewrite the day’s plan while hundreds of cast and crew wait idle.

We realized that while generative AI has revolutionized screenwriting and concept art, production operations remain trapped in fragmented spreadsheets, manual phone calls, and chaotic set-side decision-making. We set out to build CineAI an autonomous Agentic Assistant Director that coordinates department readiness in real-time and computes instant, viable contingency plans the moment a blocker arises.

🎬 What It Does

CineAI is an AI-orchestrated production control center that turns reactive firefighting into proactive set management:

Intelligent Screenplay & Call Sheet Ingestion: Leverages IBM Docling MCP to ingest and parse complex, unstructured production documents screenplays, scene breakdown sheets, and safety riders transforming raw PDFs into structured operational metadata for our agents.
Scene Resolution & Breakdown: The AD orchestrator (ad_multi_agent) resolves any requested scene against ground-truth production databases, extracting scene requirements, time-of-day constraints, gear setups, and stunt levels.
Parallel 5-Department Feasibility Check: Simultaneously coordinates five specialized AI department agents:
- 🌤️ Weather Agent: Cross-references forecast readings against rain probability and wind thresholds (e.g., golden hour outdoor shoots).
- 📍 Location Agent: Verifies location feasibility, operating hours, access constraints, and municipal permit clearance.
- 🎥 Camera Agent: Checks inventory availability across multi-camera and drone rigs against maintenance logs.
- 💥 Stunt Agent: Verifies mandatory safety checklists (harnesses, crash mats, medical team sign-offs).
- 👥 Cast Agent: Assesses mapped actor availability against daily call sheets.
Autonomous Replanning Engine: If any department returns a blocker (ready_for_shooting = false), CineAI automatically invokes the Replan Agent. The agent scans the production database, filters candidate scenes, evaluates equipment and location compatibility, and provides ranked schedule swaps with clear rationales.
Interactive Production War Room UI: Displays a live visual multi-agent workflow graph, 24 FPS SMPTE timecode clock, interactive department telemetry cards, raw agent JSON inspectors, and one-click "Approve Swap" schedule updates.

🛠️ How We Built It

Multi-Agent Orchestration: Built with Google ADK (Agent Development Kit) and powered by Gemini 2.5 Flash on Google Cloud Vertex AI.
IBM MCP Tooling & Document Understanding:
- Integrated IBM Docling MCP (docling-mcp) via Google ADK's McpToolset over stdio to enable high-accuracy document parsing, table extraction, and structural understanding of production call sheets and script supervisor notes.
- Configured optional IBM watsonx.data Document Library Retrieval MCP (ibm-watsonxdata-dl-retrieval-mcp-server) for enterprise RAG over production archives.
Agent Architecture:
- ad_multi_agent (Root Orchestrator): Parses natural language queries, resolves scene IDs, integrates IBM Docling MCP tools, and synthesizes overall briefings.
- production_workflow (ParallelAgent): Executes concurrent department checks asynchronously across Weather, Location, Camera, Stunt, and Cast.
- replan_agent: Triggered conditionally upon any department failure using DB retrieval tools to formulate contingency options.
Database & Fact Grounding: Backed by a relational PostgreSQL database running in Docker, housing normalized tables for scene_master, scene_breakdown, location_master, weather_requirements, camera_schedule, stunt_safety_checklist, and cast_availability. Every agent decision is grounded in real SQL facts to eliminate hallucinations.
Backend Bridge: Built using FastAPI and Uvicorn, providing dual execution modes:
- Instant DB Verification Mode (<50ms): Direct deterministic tool execution for rapid exploration.
- Live Streaming Mode: Server-Sent Events (SSE) streaming real-time Gemini agent reasoning and function calls directly to the client.
Frontend / UI: Developed with semantic HTML5, modern Vanilla CSS (dark glassmorphism, glowing telemetry, responsive typography using Plus Jakarta Sans and JetBrains Mono), and reactive ES6 JavaScript.

⚡ Challenges We Ran Into

Bridging MCP with Multi-Agent Runtimes: Integrating IBM Docling MCP across stdio transport within the Google ADK orchestrator required careful lifecycle management and timeout tuning to ensure document parsing ran smoothly alongside real-time DB queries.
Preventing LLM Hallucinations in Production Logistics: In physical filmmaking, an LLM cannot "assume" a camera exists or that an actor is available. We solved this by strictly grounding every agent in explicit DB tool contracts (check_camera_for_scene, check_weather_for_scene, etc.), ensuring the model reasons only over verifiable database rows.
Coordinating Parallel Agent Failures Gracefully: When multiple departments fail at once (e.g., both weather and camera blocked), synthesizing a single coherent replan without race conditions was challenging. We structured ProductionWorkflow to aggregate all department state deltas before delegating to the replan_agent.
Real-Time Streaming to the UI: Piping asynchronous ADK JSONL CLI outputs and multi-agent events through FastAPI SSE into a live frontend required robust line buffering and event parsing.

🏆 Accomplishments That We're Proud Of

Pioneering Open MCP Integration: Seamlessly combined IBM Docling MCP tools with Google ADK multi-agent orchestrators, showcasing cross-ecosystem agent interoperability for complex production document parsing.
True Multi-Agent Parallelism: Successfully orchestrated 5 independent department agents running concurrently, reducing full production check times from minutes of manual phone calls to sub-second evaluations.
Dynamic Contingency Replanning: When a scene like SC03 (Camera in Maintenance) is flagged, the system autonomously identifies SC01 or SC02 as ideal swaps, outlines the required call sheet modifications, and calculates compatibility scores.
Cinematic War Room Aesthetic: Created an intuitive, high-tech interface that feels right at home in a high-budget film production trailer or director's monitor rig.
Dual-Speed Architecture: Supported both instant <50ms deterministic verifications for instant UI responsiveness and live Vertex AI streaming for deep agentic reasoning.

📚 What We Learned

The Power of the Model Context Protocol (MCP): Utilizing open MCP servers like IBM Docling allows specialized, heavyweight document processors to plug directly into modular agent frameworks without rewriting complex conversion pipelines.
Domain Specialization Beats Generalism: Dividing production checks into discrete micro-agents (Camera, Stunt, Weather, Cast, Location) yielded vastly higher accuracy and better explainability than prompting a single monolithic LLM.
Grounded Tool Calling is Essential: Providing strict Pydantic schemas and deterministic database tools turns generative models into dependable operational decision engines.
Respecting Physical Set Constraints: Software designed for creative production must respect real-world film physics union hours, daylight windows, and safety protocols.

🚀 What's Next for CineAI

Expanded IBM Docling & watsonx Pipeline: Deepen real-time PDF diffing with Docling to automatically detect revision marks in newly issued shooting scripts and propagate changes across the 5 department agents.
Live MCP Integrations: Connect live weather radar APIs (e.g., OpenWeather MCP) and municipal city permit portals to detect weather anomalies and permit approvals in real time.
Automated Call Sheet & WhatsApp/SMS Dispatch: One-click generation of Hollywood-standard PDF call sheets with automated SMS/WhatsApp alerts to cast and crew leads whenever a schedule swap is approved.
Budget Impact Simulator: Real-time cost calculation showing the exact dollars saved or equipment rental variance resulting from each recommended schedule swap.