-
-
Choose a fault to simulate
-
Two agents diagnose independently: video and telemetry
-
Safety Gate: eight checks before any action
-
Recovery verified in video and metrics
-
Incident closed and reported
About the Project
When something breaks behind the stream, every second matters.
We were inspired by a common, recurring incident across media and entertainment and the opportunity to build a reusable, scalable solution that minimizes viewer impact, downtime, and business loss across the industry.
Rather than trying to automate every media-operations scenario, we focused on one realistic incident and demonstrated the complete lifecycle end to end — from detection to verified recovery.
Try it here: open the demo, click "Try it now", choose Encoder Failure, and watch all 10 steps run to a verified recovery in under a minute. Expand any step to see its evidence, then download the incident report.**
1. The Problem
- In live streaming, detecting an incident is not the hardest part — understanding why it happened and fixing it quickly is — to minimize viewer impact, downtime, and business loss.
Traditional monitoring can tell engineers that something is wrong, but not:
- What is the viewer actually experiencing?
- What is the root cause?
- What is the safest remediation?
- Did the remediation actually restore the stream?
An on-call engineer may need to manually inspect:
- Infrastructure telemetry
- Logs and metrics
- The actual video stream
- Possible root causes
- Remediation options
- Post-fix recovery
This can take more than 15 minutes, resulting in viewer frustration, drop-off, and potential revenue loss.
Our vision
Don't just detect the incident. Understand it, fix it safely, and prove that it is fixed.
That vision became MediaOps CoPilot — an autonomous first responder for live media infrastructure.
2. What We Built
MediaOps CoPilot is an autonomous, multi-agent incident-response platform for live streaming.
It follows a complete closed-loop lifecycle:
Detect → Diagnose → Decide → Gate → Execute → Verify → Learn
For our hackathon demo, we focused on a realistic encoder failure affecting a live stream. The live demo covers three fault types (encoder failure, encoder overload, network degradation) and four bounded actions: restart encoder, reduce profile, switch source and failover.
3. How It Works
① Detect
- A deterministic monitoring layer continuously monitors infrastructure telemetry.
- It looks for sustained anomalies, rather than reacting to temporary spikes.
- This helps prevent false incidents.
② Diagnose
Two specialised AI agents investigate the incident in parallel:
Vision Agent
- Analyzes the actual video.
- Identifies viewer-visible issues such as:
- Freezing
- Black frames
- Macroblocking
- Video degradation
Infra Agent
- Queries infrastructure telemetry through the Grafana MCP server, such as:
- CPU
- Frame rate
- Packet loss
- Encoder health
Together they answer two critical questions:
What does the viewer see? What does the infrastructure say?
- The incident is confirmed only when the evidence corroborates the failure.
③ Decide
A Remediation Agent combines:
- Diagnostic findings
- Current incident context
- Similar previously resolved incidents from the Knowledge Base
It recommends one action from a tested set of safe remediation actions.
Important: The AI can recommend an action, but cannot execute it directly.
④ Gate — Safe Autonomy
The Safety Gate is a deterministic, fail-closed control layer.
Before anything is executed, it checks:
- Is the action approved?
- Is the target component correct?
- Is the action within the allowed remediation set?
If anything is invalid or uncertain:
BLOCK.
The gate runs eight checks before any action: allow list, incident match, action-to-fault compatibility, approved target, evidence confidence (0.8 minimum), blast radius, action budget (2 per incident) and cooldown. Any failed check blocks the action.
This creates a clear separation:
AI = Judgment Deterministic code = Authority
⑤ Execute
- Approved actions are sent to the Control Plane.
- Remediation is executed idempotently.
- Duplicate incidents or events cannot accidentally trigger duplicate fixes.
⑥ Verify
We don't assume that a successful command means successful recovery.
MediaOps CoPilot independently verifies recovery using:
- Infrastructure telemetry
- Actual video quality
The incident is closed only after sustained recovery is confirmed.
“It ran” is not the same as “it recovered.”
⑦ Learn
- The incident, evidence, remediation, and verified outcome are stored as operational memory.
- Similar future incidents can retrieve this knowledge and benefit from previous resolutions.
This creates a continuous:
Incident → Resolution → Learning → Better Future Response
loop.
4. Architecture & Technology
We built MediaOps CoPilot as an multi-agent system using Google Cloud and Google ADK.
Core technologies
- Google ADK — multi-agent orchestration and workflow management
- Gemini 2.5 Flash / Vertex AI — multimodal reasoning
- Cloud Run — hosted application deployment
- Firestore — incident state, audit trail, and operational memory
- Vertex AI Embeddings — semantic retrieval of similar incidents
- Grafana + Prometheus — monitoring and observability. The Infra Agent reads telemetry through the official Grafana MCP server (grafana/mcp-grafana, read-only; local run, the hosted demo uses the Prometheus fallback), so its diagnosis is built on live Grafana queries.
- FFmpeg — real video workload and fault simulation
- Safety Gate — deterministic authorization
- Control Plane — guarded remediation execution
- Slack — operational notifications
- Testing — 90 automated tests cover the safety gate, orchestrator, control plane, verification, reporting and knowledge-base write-back
5. Our Key Innovation
The most important architectural decision was:
AI supplies judgment. Deterministic code verifies and supplies authority.
Instead of allowing an LLM to directly control infrastructure:
AI Agents → Investigate → Reason → Recommend
Safety Gate → Validate → Authorize / Block
Control Plane → Execute
Verification → Prove recovery
This gives us agentic intelligence with controlled execution.
6. What We Learned
- Think beyond one incident: Identify recurring problems and their scale and impact across the media and
entertainment industry. - Build for scale and reuse: Turn a specific solution into a reusable capability that can benefit many.
- Agentic AI is about the workflow — The value lies in orchestration, guardrails, execution, verification, and learning.
- A powerful LLM alone is not enough to create an autonomous system.
- The real engineering challenge is designing: *Architecture decisions and trade-offs — multi-agent vs. single-agent, LLM selection, cost vs. quality vs. latency, and Responsible AI safeguards.
- Workflow orchestration
- State management
- Guardrails
- Failure handling
- Verification
- Learning loops
Different evidence needs different agents
- Video and infrastructure telemetry provide different perspectives.
- Specialized agents can investigate each evidence source independently.
- Corroborating their findings creates stronger incident confidence.
AI should recommend — not automatically have authority
- We learned to treat AI output as a proposal, not an instruction.
- This allows us to use agentic reasoning while keeping execution deterministic and controlled.
Recovery must be proven
- A successful API response or restart command does not necessarily mean the viewer experience has recovered.
- True recovery requires independent verification.
7. Key Challenges
Building a system that is both autonomous and predictable was our biggest challenge.
We had to solve for:
- Sustained anomaly detection vs. transient spikes
- Duplicate incident prevention
- Duplicate remediation prevention
- Parallel agent execution
- Workflow state management
- Conflicting evidence between agents
- Restricted and approved remediation actions
- Fail-closed safety controls
- Idempotent execution
- Auditable incident history
- Reusable operational memory
Hackathon trade-off
Rather than attempting to automate every possible media-operations scenario, we focused on one complete, realistic incident lifecycle end to end.
That allowed us to demonstrate the full loop:
Failure → Detection → Diagnosis → Decision → Safe Remediation → Verified Recovery → Learning
Current limits: the stream and telemetry come from a simulator, the demo runs a single encoder, and the hosted version reads Prometheus directly.
8. Impact
Our goal was to demonstrate measurable improvement over traditional manual incident response.
Results from our testing
- ⚡ 44 seconds average autonomous recovery (average of 5 test runs)
- ⏱️ more than 15 minutes typical manual response
- ⏱️ What this means for viewers (illustrative example): on a live stream with 50,000 concurrent viewers, a 15-minute manual response means about 750,000 viewer-minutes of broken video. At 44 seconds, that drops to about 37,000, a reduction of roughly 95%.
- 📹 Recovery verified using actual video + infrastructure telemetry
- 🧾 Every incident creates an auditable record
- 🧠 Resolved incidents become reusable operational knowledge
The result is not simply an AI chatbot.
It is a closed-loop autonomous incident-response system that can:
Investigate → Decide → Safely Remediate → Verify → Learn
9. What's Next
We started with live streaming because the impact is highly visible:
When something breaks behind the stream, every second matters.
But the architecture can extend beyond a single encoder failure or streaming scenario.
Our broader vision is an autonomous operational layer for media and entertainment that can:
Detect → Understand → Safely Act → Verify → Learn
Make It Scalable
Build MediaOps CoPilot as an API-first, multi-tenant SaaS platform that media companies can easily integrate with their existing monitoring, video, and infrastructure systems.
Tenant → API Gateway → MediaOps Orchestrator → Agents → Safety Gate → Control Plane → Verification
Scalability comes through:
- Multi-tenancy — isolated data, policies, configurations, and knowledge per company
- API-first integration — plug into existing monitoring and infrastructure without replacing them
- Reusable agents and workflows — extend the same capability to new incident types
- Event-driven architecture — scale incident processing independently as volume grows
- Configurable safeguards and remediation — adapt policies to each company's environment
This enables MediaOps CoPilot to evolve from a hackathon solution for one incident into a reusable, scalable operational capability across the media and entertainment industry.
Reactive incident management → Proactive, trusted autonomy
10. Additional Demo Video:
Simulation Encoder Overload: https://www.youtube.com/watch?v=WoWEl66Gbhc **
Our Guiding Principle
When something goes wrong behind the stream, the viewer shouldn't have to know.
MediaOps CoPilot Detect it. Understand it. Fix it. Prove it's fixed.
Log in or sign up for Devpost to join the conversation.