ReelWarden

ReelWarden helps film studios protect deadlines by turning live render metrics, logs, and traces into clear AI-guided decisions: what is at risk, what to approve, and proof that recovery worked.

Comment

Inspiration

Film productions face fixed delivery deadlines while several shows compete for the same render capacity. Infrastructure teams can see queue depth, worker health, logs, and traces, but producers need a simpler answer: will we make the deadline, what is at risk, and what should we do now?

ReelWarden was inspired by the gap between technical observability and production decision-making. We wanted to turn low-level render-farm signals into a clear, evidence-backed recommendation that a producer can understand and approve.

What it does

ReelWarden helps studios manage two productions sharing a fixed pool of 32 render workers. It identifies capacity conflicts, compares delivery plans, and translates active incidents into production impact.

Using Google ADK and Gemini 3.8 Flash, the agent investigates live Grafana metrics, logs, and traces through the official Grafana MCP server. It explains what happened, which scene and delivery milestone are affected, the available recovery options, their cost and risk tradeoffs, and the recommended action.

The AI does not execute changes on its own. A producer must approve the recommendation, and ReelWarden then verifies recovery using fresh Grafana evidence.

How we built it

ReelWarden combines a React dashboard, a Python render-farm simulator, Grafana, Prometheus, Loki, Tempo, Google ADK, Gemini on Vertex AI, and the official Grafana MCP server.

The simulator generates production events, render metrics, logs, traces, queue depth, worker allocation, and delivery forecasts. A deterministic calculator owns schedule, cost, capacity, and risk calculations. Gemini uses those validated facts to explain the situation and select from valid options.

The dashboard supports portfolio planning, incident investigation, alternative plans, what-if calculations, human approvals, recovery verification, conversation, and audit history. SQLite stores investigations, approvals, messages, execution records, and verification results.

The project is hosted on Google Compute Engine. Gemini uses the VM’s attached service account for Vertex AI authentication, avoiding stored service-account keys.

Challenges we ran into

The biggest challenge was making the agent useful without allowing it to become an untrusted decision-maker. We separated observed Grafana evidence, calculator forecasts, and AI-written explanations. The agent can only select currently valid options, and every execution requires human approval.

Operational telemetry also arrives at different times. Metrics, logs, and traces may not be available simultaneously after an event. We added evidence retries, live progress, and explicit failure states so the app never claims a verified live result without the required evidence.

We also had to prevent raw JSON, tool output, and internal agent events from appearing in the conversation. We created a presentation boundary that parses structured responses and renders only validated, production-friendly content.

Accomplishments that we're proud of

We built a complete workflow from delivery risk to verified recovery. The app starts with two productions competing for capacity, recommends a grounded allocation change, requires approval, and verifies the resulting worker allocation.

For incidents, ReelWarden can investigate six distinct failure scenarios, including GPU memory pressure, worker loss, queue surge, storage pressure, network latency, and corrupted assets. Each scenario is backed by metrics, correlated logs, and traces in Grafana.

We are especially proud that a forced failed recovery triggers a fresh reassessment, produces a different valid recommendation, and requires a new approval. This keeps the experience honest instead of presenting a false success.

What we learned

We learned that observability becomes much more useful when it is connected to a real decision. A Grafana dashboard can show what is happening, but producers need context, delivery impact, alternatives, tradeoffs, and a clear next step.

We also learned that trust in agentic systems comes from constraints and verification. Gemini is effective at investigating evidence and communicating tradeoffs, while deterministic systems should own calculations and people should retain authority over execution.

What's next for ReelWarden

Next, we want to connect ReelWarden to real render-management systems and studio scheduling data instead of a simulated render farm. We would add user authentication, role-based approvals, HTTPS, production-grade audit controls, and team notifications.

We also want to expand from incident response into proactive planning. ReelWarden could forecast capacity risks weeks before a deadline, compare vendor-render capacity against internal capacity, and help producers balance budget, delivery confidence, and creative priorities across an entire studio slate.

Built With

Share this project:

Updates

Submission history