Volume Ops

Inspiration

On a virtual production stage, an infrastructure problem can become a filmmaking problem in seconds. A render failure can interrupt an LED background, stop a take, and leave an entire crew waiting. Monitoring dashboards show technical problems, but film crews need practical answers: Can we keep shooting? What is at risk? What should we do next? I built Volume Ops to bridge that gap.

What it does

Volume Ops is an agentic on-call system for virtual production LED stages. It investigates Grafana telemetry and turns technical findings into practical guidance for film crews. Each investigation provides:

• A likely root cause supported by metrics or logs.
• An explanation of the potential impact on filming.
• A production cost estimate calculated from explicit rules.
• Clear recovery recommendations. The interface displays hypotheses, queries, evidence, and rejected explanations as the investigation unfolds.

How we built it

Volume Ops uses Google ADK, Gemini models, and Grafana Cloud, connected through the official Grafana MCP server. The investigation follows four stages:

1. Triage: Forms initial hypotheses from an alert.
2. Investigator: Queries metrics and logs, tests hypotheses, and expands the search window when necessary.
3. Scorer: Uses deterministic Python rules to assess supported findings and calculate production cost estimates.
4. Brief: Translates the findings into plain language for the crew. The scorer contains no language model. Cost estimates come from configured rules and arithmetic, and findings without supporting evidence are dropped. Server-Sent Events stream investigation updates to the interface. The MCP tool selection is narrowed to eight relevant tools to keep investigations focused.

Challenges we ran into

The hardest challenge was separating when a failure appeared from when its cause occurred. Recent errors can explain the symptoms while the triggering change happened hours earlier. The investigator needed to search beyond the immediate alert window. Another challenge was keeping model-generated explanations grounded in evidence. An evidence requirement and a deterministic Python scorer prevent an unsupported explanation from determining the final assessment. Limited model-call quotas also constrained testing. This required carefully planned investigation runs and reusable demo recordings. Finally, technical findings had to be translated into language a film crew could act on without losing the connection to the underlying evidence.

Accomplishments that we're proud of

Volume Ops connects observability data to the practical needs of a film crew. The project combines a visible investigation process, evidence requirements, deterministic cost calculations, and clear recovery guidance in one interface. One particularly useful feature is the visible rejection of hypotheses. Users can see which explanations were considered, why they were ruled out, and how the investigation reached its conclusion.

What we learned

An effective agent needs ways to challenge its first explanation. Checking unaffected systems and expanding the search window can be just as important as finding an error. The project also showed the value of combining language models with deterministic code. Models help investigate and explain; explicit rules handle calculations and evidence requirements. Transparency matters to the user experience. Showing the investigation helps people assess the recommendation.

What's next for Volume Ops

The next steps are to connect Volume Ops to real stage telemetry, validate cost assumptions with production teams, and evaluate the system against a wider range of incidents. Future development would also introduce operator-approved recovery workflows, allowing crews to review proposed actions before they change stage infrastructure.

Built With

• events
• google-adk
• google-gemini
• grafana
• grafana-cloud
• model-context-protocol-(mcp)
• python
• server-sent