Render Farm Incident Copilot

Inspiration

VFX and media-production teams depend on render farms to deliver shots on time. When a render queue is saturated, supervisors must decide quickly whether the issue is a known software problem, a capacity problem, or an operational incident. I built Render Farm Incident Copilot to turn that stressful manual investigation into an auditable, evidence-backed workflow.

What it does

Render Farm Incident Copilot investigates a render-farm incident using specialized agents.

The Research Agent uses Parallel Search and Task APIs to collect live external research. The Metrics Analyst uses the official Grafana MCP server to retrieve live Prometheus telemetry and firing Grafana alerts. The Verification Agent cross-references the evidence.

The system reaches a Human Approval Gate only when it has live Parallel citations, live Grafana metrics, and a firing Grafana alert. It can recommend adding GPU capacity, but a human must approve before any infrastructure action.

How I built it

I built the frontend with React, TypeScript, and Vite, and the agent backend with Python and FastAPI.

The demo render-farm workload exposes instrumented metrics. Prometheus scrapes those metrics and remote-writes them to Grafana Cloud. The backend reads the resulting telemetry through the official Grafana MCP server.

The public Render deployment runs its own Prometheus collector, so it generates clearly labeled demo telemetry independently of a developer laptop. Gemini API and Google ADK support agent planning and synthesis, while Parallel provides live research evidence.

Challenges I ran into

The hardest challenge was making the demo trustworthy instead of merely visually convincing.

I needed to ensure that the app never substituted local values or invented citations when Grafana, Parallel, or Gemini was unavailable. I also needed the public deployment to keep producing telemetry without relying on a local machine.

I solved this with a real Prometheus remote-write pipeline, a real Grafana-managed firing alert, live MCP queries, and a deterministic evidence gate.

Accomplishments that I'm proud of

• Built a public deployed application with live evidence retrieval.
• Connected Parallel Search and Task APIs to the investigation workflow.
• Connected Grafana Cloud through the official Grafana MCP server.
• Retrieved live GPU utilization, render queue depth, and a firing Render queue saturation alert.
• Made the hosted Render deployment generate its own telemetry.
• Enforced a Human Approval Gate that blocks recommendations when required evidence is missing.

What I learned

I learned that agentic systems are more trustworthy when the model is not the final authority.

Gemini can help plan, summarize, and synthesize information, but the approval decision should be enforced by deterministic application logic. I also learned how to combine Prometheus, Grafana Cloud, MCP, and research APIs into one verifiable workflow.

What's next for Render Farm Incident Copilot

Next, I want to connect real render-management systems, production schedules, and asset metadata. I would add role-based approval workflows, incident history, notification routing, and cost-aware scaling recommendations.

The goal is to evolve the copilot from a live incident investigator into an operational command center for VFX and media-production infrastructure.

Built With

• adk
• api
• cloud
• context
• docker
• fastapi
• gemini
• google
• grafana
• mcp
• model
• parallel
• prometheus
• protocol
• python
• react
• render
• typescript
• vite