Studio Ops Copilot

Inspiration

Film and television production is governed by unforgiving delivery deadlines: morning executive dailies, film festival DCP handoffs, and live global premieres. Behind these deadlines are massive, complex technical pipelines:

GPU render farms running Blender, Maya, and Houdini. High-throughput transcode clusters crunching 4K/8K ProRes 4444 into AV1 and SMPTE DCPs. Low-latency live broadcast ingest paths streaming global premiere events. These systems produce thousands of metrics, logs, and alerts in Grafana Cloud every minute. But when an overnight render queue backs up at 3:00 AM or a premiere livestream drops frames, the people who need to know first—post-production supervisors, VFX coordinators, and live broadcast leads—are not Site Reliability Engineers. Staring at raw PromQL queries and time-series panels under immense deadline pressure is overwhelming.

We built Studio Ops Copilot to eliminate this bottleneck: giving creative and operational crews a conversational, natural-language interface directly into their production infrastructure.

What it does

Conversational Studio Operations: Crew members ask questions in plain English:

"Why is the overnight render queue backed up?" "Is the premiere livestream healthy right now?" "Show me error rates on the 4K transcode pipeline." "List all firing alerts across our pipelines." Real-time Grafana MCP Integration: The agent actively connects to the official Grafana Model Context Protocol (MCP) server (grafana/mcp-grafana) using the Python mcp SDK (ClientSession over stdio/SSE) to execute:

query_prometheus: PromQL queries for queue depth, GPU VRAM usage, and frame render duration. query_loki: LogQL queries to isolate CUDA OOM crashes, disk exhaustion, and video packet loss. list_alerts: Instant inspection of firing Alertmanager rules. search_dashboards: Discovery of production dashboards and panel deep-links. Plain-English Studio Incident Briefs: Powered by Google Gemini, the agent synthesizes raw telemetry into an executive brief structured in 3 parts:

🎬 Production Impact Status: The real-world cinema effect (e.g. Sequence 14 / Shot 42 final composite delayed by 3.8 hours). 📊 Telemetry & Root Cause: Plain-language explanation of the underlying infrastructure failure (e.g. 18 worker nodes failed due to CUDA Out of Memory on 8K volumetric passes). 🛠️ Recommended Action: 1-3 immediate operational steps for the crew (e.g. Redirect priority queue to 48GB VRAM node pool, purge uncompressed scratch cache). Realistic Studio Pipeline Telemetry: Includes a built-in synthetic telemetry generator simulating:

VFX Render Farm: 64 GPU worker nodes, queue backlog, CUDA OOM failure rates, VRAM saturation. Premiere Livestream: SRT ingest bitrate, dropped frames, viewer concurrency, audio-video drift. Post-Mastering Queue: DCP transcode throughput and chunk errors.

How we built it

AI & Reasoning Layer: Google Gemini Flash (gemini-3.5-flash / gemini-flash-latest) as the core cognitive engine. Google Agent Development Kit (google-adk) and the official Google GenAI SDK (google-genai) for autonomous tool calling and structured function execution. Custom system persona tailored specifically to cinema technical operations and post-production workflows. Live Runtime Environment: The hosted deployment runs live Google Gemini model inference alongside live Grafana Cloud MCP tool executions. Observability Layer: Official Grafana MCP Server (grafana/mcp-grafana) using the Python mcp SDK (ClientSession) supporting both stdio and sse transports with dynamic tool discovery via list_tools(). High-fidelity fallback MCP provider ensuring instant zero-friction demonstrations for local offline testing. Standard Prometheus /metrics exposition endpoint for scraping by Grafana Cloud Agent / Alloy. Backend & Control Room Web Application: FastAPI backend in Python providing conversational streaming, scenario state toggling, and telemetry endpoints. Ops Room Control Center UI designed specifically for engineers under pressure: 3-column command center with pipeline rail, incident brief feed, and ticking monospace telemetry strip. Deployment & Cloud Infrastructure: Fully containerized with Docker bundling the official grafana/mcp-grafana binary. Hosted and continuously deployed on Render.com (with Google Cloud Run deployment scripts also provided via cloudbuild.yaml and deploy.sh). Powered by Google Gemini Flash via API keys from Google AI Studio.

Challenges we ran into

During the development of Studio Ops Copilot, our key technical findings and architectural learnings included:

The Power of Standardized MCP vs. Custom REST Integrations: Prior to MCP, integrating telemetry into LLMs required writing brittle custom HTTP clients that scraped individual Grafana REST endpoints and hardcoded schema assumptions. By implementing the official Model Context Protocol (mcp.ClientSession to grafana/mcp-grafana), dynamic tool discovery (session.list_tools()) allowed Google Gemini to inspect and adapt to available telemetry tools automatically. This reduced maintenance friction and eliminated schema drift.

Async Protocol Lifecycle in Production Container Environments: Managing persistent stdio and SSE transport connections across asynchronous web frameworks (FastAPI) required establishing a single lifecycle-managed ClientSession in the application lifespan rather than opening and closing subprocesses per user request. This decreased latency per query from ~2.8s to <400ms.

Bridging Creative Terminology with SRE Telemetry: Raw telemetry (e.g. PromQL: sum(rate(container_cpu_usage_seconds_total...))) is unintelligible to post-production supervisors facing an executive screening. Grounding Gemini's system persona with domain-specific cinema concepts (e.g., volumetric render passes, ProRes 4444 XQ mastering queues, SRT ingest jitter) transformed cryptic error codes into actionable operational decisions with estimated delivery impacts.

Zero-Friction Fallback Architecture: Building a dual-layer architecture—connecting to live Grafana Cloud and Gemini when credentials exist, while maintaining an authentic high-fidelity simulation and synthetic Prometheus exposition endpoint (/metrics)—ensures evaluation judges and testing environments can verify functionality without setup friction.

What we learned

During the development of Studio Ops Copilot, our key technical findings and architectural learnings included:

The Power of Standardized MCP vs. Custom REST Integrations: Prior to MCP, integrating telemetry into LLMs required writing brittle custom HTTP clients that scraped individual Grafana REST endpoints and hardcoded schema assumptions. By implementing the official Model Context Protocol (mcp.ClientSession to grafana/mcp-grafana), dynamic tool discovery (session.list_tools()) allowed Google Gemini to inspect and adapt to available telemetry tools automatically. This reduced maintenance friction and eliminated schema drift.

Async Protocol Lifecycle in Production Container Environments: Managing persistent stdio and SSE transport connections across asynchronous web frameworks (FastAPI) required establishing a single lifecycle-managed ClientSession in the application lifespan rather than opening and closing subprocesses per user request. This decreased latency per query from ~2.8s to <400ms.

Bridging Creative Terminology with SRE Telemetry: Raw telemetry (e.g. PromQL: sum(rate(container_cpu_usage_seconds_total...))) is unintelligible to post-production supervisors facing an executive screening. Grounding Gemini's system persona with domain-specific cinema concepts (e.g., volumetric render passes, ProRes 4444 XQ mastering queues, SRT ingest jitter) transformed cryptic error codes into actionable operational decisions with estimated delivery impacts.

Zero-Friction Fallback Architecture: Building a dual-layer architecture—connecting to live Grafana Cloud and Gemini when credentials exist, while maintaining an authentic high-fidelity simulation and synthetic Prometheus exposition endpoint (/metrics)—ensures evaluation judges and testing environments can verify functionality without setup friction.

What's next for Studio Ops Copilot

Automated Self-Healing Remediation: Giving the agent supervised execution capabilities to trigger Kubernetes worker restarts or route jobs via Deadline / OpenCue APIs. Voice-Activated Ops in the Edit Suite: Integrating Google Cloud Speech-to-Text and Text-to-Speech so editors in dark grading suites can check render health hands-free. Predictive Render Queue Estimation: Using Gemini multimodal capabilities to inspect storyboard complexity and forecast render times before artists submit shots to the farm.

Built With

• google
• grafana