-
-
After human-approved failover, Stagehand verifies 12.3/12.5 ms frame times and 2.6 ms LED sync, safely below both thresholds.
-
Grafana exposes inc-1042: render-3 at 29.0 ms and 98% GPU pressure, LED sync at 14.0 ms, plus correlated Loki failures.
Inspiration
Virtual-production failures cross creative and technical boundaries. A delayed render frame can look like LED-wall synchronization drift, camera-tracking trouble, network congestion, or GPU exhaustion. During a take, a production supervisor has to correlate dashboards and logs while an expensive crew waits, then decide whether the evidence supports continuing or isolating part of the render cluster.
My experience working in observability at Sumo Logic and AppDynamics shaped Stagehand’s approach: operational AI should correlate evidence, test alternative explanations, state uncertainty, and preserve human authority over consequential actions.
What it does
Stagehand turns that investigation into a bounded, evidence-grounded workflow.
A deterministic virtual-stage simulator creates GPU pressure on render-3 and exports fresh metrics, logs, and traces to Grafana Cloud. Gemini, orchestrated by Google ADK, queries the official read-only Grafana MCP server, correlates Prometheus metrics with Loki events, checks tracking and network telemetry as counter-evidence, and recommends a production action.
Gemini cannot execute remediation. A human supervisor must explicitly approve the active incident before Stagehand performs the simulated failover. Stagehand then verifies recovery from fresh telemetry: the remaining render nodes must stay below the 16.7 ms frame budget and LED synchronization must return below 8 ms.
How we built it
Gemini on Vertex AI reasons over live operational evidence and produces the incident recommendation. Google ADK defines and runs the agent, manages the incident-scoped session, connects Gemini to Grafana MCP, and streams investigation events. Grafana Cloud stores the OpenTelemetry signals and supplies Prometheus metrics and Loki logs. Official Grafana MCP gives Gemini controlled read-only evidence queries. It is launched with write operations disabled. FastAPI owns the deterministic simulator, protected control endpoints, approval boundary, and recovery state machine. Google Cloud Run hosts the combined application. Secret Manager and IAM protect Grafana and operator credentials.
The telemetry path and action path are deliberately separate. The simulator sends evidence to Grafana; Gemini can only read that evidence. The approved action travels through FastAPI and is never exposed as an agent tool.
The demonstrated incident
GPU memory pressure drives render-3 to 29.0 ms frame time and 98% GPU memory utilization while LED synchronization reaches 14.0 ms. Grafana Loki contains the correlated gpu_allocation_failed event. Healthy tracking and network latency help Gemini reject two plausible alternative causes.
Gemini recommends isolating render-3 and explicitly stops at the approval boundary. After human approval, Stagehand removes that node from the active pool. Recovery is declared only after render-1 and render-2 return to 12.3 ms and 12.5 ms and LED synchronization returns to 2.6 ms.
Challenges we ran into
The hardest part was preserving a real agentic investigation without giving the model uncontrolled operational authority. We separated reasoning from execution, restricted Grafana MCP to read-only tools, bound approval to the active incident, and rejected stale or duplicate confirmations.
We also had to make live Grafana ingestion, MCP read-back, browser authorization, Cloud Run IAM, and ADK streaming work together without leaking credentials or exposing model-backed endpoints to anonymous traffic.
Accomplishments that we're proud of
A complete incident, investigation, approval, failover, and recovery loop running on Cloud Run. Runtime Grafana usage through the official MCP server, not a README-only integration. Live Prometheus and Loki evidence with production context carried across stage, scene, take, incident, and render node. A human-in-the-loop boundary that Gemini cannot bypass. A public read-only judge console with protected Gemini, ADK, and mutation routes. 35 passing automated tests, deterministic safety-contract evaluations at 1.0, and a Vertex response-quality evaluation averaging 5/5.
What we learned
Operational AI is more credible when it explains what evidence supports a decision, what evidence argues against it, and what remains unknown. Determinism and human authority do not make an agent less useful; they make its role legible enough to trust during a costly production incident.
What's next
Stagehand currently models one GPU-pressure incident on one virtual stage. Next steps include additional failure modes, persistent multi-stage sessions, incident replay, richer Grafana deep links, and adapters for real render-farm and virtual-production control systems.