Showrunner AI

Never Drop a Frame.

Comment

Inspiration

In feature film and episodic virtual production, render farms are the ultimate bottleneck. A single hero shot overflowing its frame buffer with high-density geometry or uncompressed 8K textures can trigger a fatal CUDA_ERROR_OUT_OF_MEMORY crash on a GPU node. When this happens overnight, pipelines stall, deadlines slip, and technical directors spend hours sifting through gigabytes of raw logs. We built Showrunner AI to serve as an autonomous, 24/7 Technical Director—monitoring cluster telemetry, isolating failure signatures, and resolving crashes before humans even notice.

What it does

Showrunner AI is a closed-loop site reliability engineering (SRE) agent designed specifically for 3D animation and virtual production clusters running engines like Unreal Engine and Blender.

Autonomous Cluster Health Auditing: Queries Prometheus metrics across nodes to monitor real-time GPU load, virtual memory headroom, and queue status.

Grounded Telemetry Correlation: Never guesses or executes blind reboots. When metric spikes or node drops occur, the agent queries Grafana Loki log streams to isolate the exact stack trace (e.g., CUDA OOM or process deadlock).

Surgical Remediation Routing: Dynamically executes the least disruptive fix: Applies Arnold/Unreal tiled-memory rendering switches to lower VRAM allocation on hero shots without discarding active progress. Executes worker process restarts for hung threads and deadlocks. Triggers isolated hypervisor reboots only when complete node failure occurs.

Real-Time Studio Command Center: Provides visual telemetry cards, interactive director prompt console, live incident logs, and timeline resolution feeds.

How we built it

Grafana Labs Track & MCP Runtime Integration

Showrunner AI actively relies on the official grafana/mcp-grafana server (the official Grafana Cloud MCP server) at runtime to inspect observability data and remediate infrastructure failures.

How Grafana Cloud MCP is Used at Runtime:

  1. Runtime Telemetry Stream: Our worker nodes ship Prometheus metrics (GPU VRAM, frame render latencies) and Loki logs (CUDA crash tracebacks, asset timeouts) directly to Grafana Cloud.
  2. Model Context Protocol (MCP) Execution: When an incident occurs, our agent—built on Google Cloud Vertex AI—connects to the official grafana/mcp-grafana MCP server over JSON-RPC.
  3. Multi-Source Diagnostics via MCP Tools: The agent invokes tool calls exposed by grafana/mcp-grafana:
    • query_prometheus: Queries PromQL memory thresholds and node health states from Grafana Cloud Prometheus.
    • search_logs: Runs LogQL range queries on Grafana Cloud Loki to isolate exact error signatures (e.g., CUDA_ERROR_OUT_OF_MEMORY: Shot VFX_BATTLE_014 failed on Node 4).
  4. Correlated Remediation: Once metric spikes and log tracebacks are correlated via the Grafana MCP server, Gemini executes targeted API remediation webhooks to restore node health.

Challenges we ran into

Tool-Loop Guardrails: Designing strict operational constraints inside the system prompt so the model would consistently prioritize correlating logs before executing remediation tools.

Local Enterprise Authentication: Bypassing console API key constraints by shifting to Google Cloud Vertex AI Application Default Credentials (ADC), ensuring frictionless authentication on local developer workstations without exposed tokens.

Fast Multi-Engine State Synchronization: Structuring the in-memory cluster model to accurately mirror GPU memory footprints and verify recovery transitions immediately post-remediation.

Accomplishments that we're proud of

Achieved a verified end-to-end autonomous healing loop: taking Node 4 from a 100% VRAM saturation crash (CRASHED_OOM) back to HEALTHY (13.2 GB stabilized) with zero human intervention.

Implemented a zero-key local-to-cloud development workflow using Google Cloud IAM and ADC.

Created a clean, responsive UI dashboard that bridges high-level creative production supervision with low-level GPU systems engineering.

What we learned

Function-calling models require clear, imperative tool descriptions and deterministic return formats to make reliable SRE decisions under operational pressure.

Autonomous recovery in creative pipelines requires nuanced routing (like dynamic frame-buffer tiling) rather than blunt system reboots to prevent catastrophic progress loss.

What's next for Showrunner AI

Autodesk ShotGrid / Flow Integration: Automatically notifying production coordinators and commenting on impacted asset tasks when render parameters are modified.

Dynamic Spot Instance Orchestration: Migrating failing shots to on-demand Google Cloud GPU instances (NVIDIA A100/H100) when local workstation hardware ceilings are exceeded.

OpenTimelineIO Integration: Prioritizing cluster remediation order based on upcoming editorial cut deadlines.

Built With

Share this project:

Updates

Submission history