-
-
ShowRunner incident feed showing detected failures and the remediation actions automatically executed by the SRE agent.
-
Chaos testing simulates an encoder failure, setting the broadcast encoder state to DOWN for automated incident detection.
-
Incident reports generated by ShowRunner, capturing encoder failures, CDN outages, latency issues, and unknown hardware glitches.
-
ShowRunner dashboard displaying real-time broadcast health, CDN regions, Gemini SRE Agent status, and automated remediation metrics.
-
ShowRunner agent successfully deployed as a container on Render, with the application server running on port 8080.
Inspiration
Modern broadcast systems are expected to stay online continuously. A failure in an encoder, CDN region, or streaming pipeline can quickly become a serious service disruption.
Traditional monitoring systems are good at telling engineers that something is wrong, but they usually stop there. An engineer still has to inspect telemetry, determine the cause, choose a remediation, execute it, and document what happened.
I wanted to build something that could close this loop.
ShowRunner was inspired by the idea of applying autonomous AI agents to Site Reliability Engineering (SRE): instead of simply alerting a human, an AI agent should be able to observe the system, reason about the incident, take a bounded corrective action, verify the result, and document the incident.
What it does
ShowRunner — Autonomous SRE for Broadcast Systems is an autonomous AI SRE agent designed for a simulated broadcast infrastructure.
When an incident occurs, ShowRunner follows an:
Observe → Reason → Act → Verify → Document
workflow.
The system:
Detects failures through Grafana alerts and Prometheus metrics. Observes the affected system by retrieving recent telemetry through Grafana/Grafana MCP. Reasons about the incident using Gemini. Selects an appropriate remediation tool. Executes a bounded recovery action such as: Restarting an encoder Failing over a region Reducing bitrate Scaling CDN capacity Escalating to a human Verifies whether the system has recovered. Generates an incident report/postmortem.
This allows ShowRunner to go beyond simply reporting that something is broken and actually participate in the recovery process.
How I built it
I built ShowRunner using Python and FastAPI for the autonomous agent and API layer.
The system uses:
Google Gemini / Vertex AI for incident diagnosis and remediation decisions Grafana for observability and alerting Prometheus for collecting system metrics Grafana MCP to provide the agent with access to telemetry Docker and Docker Compose for the simulated infrastructure HTML, CSS and JavaScript for the operational dashboard
I created a simulated broadcast environment where failures can be intentionally introduced through chaos-injection scenarios.
The core workflow is:
Incident → Telemetry → Gemini Diagnosis → Remediation Tool → Recovery Verification → Postmortem
I also designed the remediation layer around predefined tools instead of allowing the AI model to execute arbitrary commands. This gives the agent a controlled set of actions it can perform.
Challenges I ran into
One of the biggest challenges was connecting the AI agent to live observability data rather than providing it with static information.
Integrating Grafana MCP with the FastAPI application required handling asynchronous MCP communication, connection management, and fallback behavior.
I also needed to make sure that one failed incident would not bring down the entire agent. I therefore added failure isolation around incident processing so individual incidents can be handled independently.
Another challenge was designing the remediation system so that Gemini could make decisions while still operating within bounded and predictable actions. Instead of giving the model unrestricted system access, I exposed specific remediation tools that it could choose from.
Finally, I had to build controlled failure scenarios that could reproduce realistic broadcast problems and demonstrate the complete failure → detection → diagnosis → remediation → recovery cycle.
Accomplishments I'm proud of
I'm proud that I built ShowRunner as an end-to-end autonomous SRE workflow, rather than just an AI chatbot or monitoring dashboard.
I can introduce a simulated failure and have the system go through:
Failure → Grafana Alert → AI Diagnosis → Automated Remediation → Recovery → Incident Report
I also implemented multiple remediation paths for different types of incidents, including encoder failures, frame-drop problems, and CDN-related failures.
Another part I'm proud of is the operational dashboard, which provides visibility into system health, incidents, remediation actions, and the state of the AI SRE agent.
Most importantly, I was able to connect generative AI with operational telemetry and controlled tools to create a practical autonomous SRE workflow.
What I learned
Building ShowRunner taught me that an autonomous AI agent is much more than connecting an LLM to a prompt.
A useful operational agent needs:
Reliable access to real-world context Structured incident information Clearly defined tools Bounded permissions Verification after taking an action Failure handling Human escalation when automation is not appropriate
My biggest takeaway was that a useful agent should not simply make a decision.
It should follow a complete loop:
Observe → Reason → Act → Verify → Document
This makes the agent's behavior more measurable, debuggable, and useful in operational environments.
What's next for ShowRunner — Autonomous SRE for Broadcast Systems
The current version uses a simulated broadcast environment. My next goal is to take ShowRunner toward more realistic infrastructure and larger operational environments.
Future improvements include:
Real broadcast infrastructure integration Correlation of multiple simultaneous incidents Historical telemetry analysis More advanced root-cause analysis Approval thresholds for high-risk remediation Automatic rollback when remediation fails Multi-region traffic management Richer incident timelines Learning from previous incident reports and postmortems Human-in-the-loop escalation for uncertain or high-impact incidents
The long-term goal is to evolve ShowRunner from a simulated broadcast incident responder into a general-purpose AI SRE capable of helping operate complex infrastructure safely and autonomousl
Log in or sign up for Devpost to join the conversation.