-
-
Choose a fault to simulate
-
Two agents diagnose independently: video and telemetry
-
Safety Gate: eight checks before any action
-
Recovery verified in video and metrics
-
Incident closed and reported
About the Project
When something breaks behind the stream, every second matters.
We were inspired by a common, recurring incident across media and entertainment and the opportunity to build a reusable, scalable solution that minimizes viewer impact, downtime, and business loss across the industry.
Rather than trying to automate every media-operations scenario, we focused on one realistic incident and demonstrated the complete lifecycle end to end — from detection to verified recovery.
Try it here: open the demo, click "Try it now", choose Encoder Failure, and watch all 10 steps run to a verified recovery in under a minute. Expand any step to see its evidence, then download the incident report.**
1. The Problem
- In live streaming, detecting an incident is not the hardest part — understanding why it happened and fixing it quickly is — to minimize viewer impact, downtime, and business loss.
Traditional monitoring can tell engineers that something is wrong, but not:
- What is the viewer actually experiencing?
- What is the root cause?
- What is the safest remediation?
- Did the remediation actually restore the stream?
An on-call engineer may need to manually inspect:
- Infrastructure telemetry
- Logs and metrics
- The actual video stream
- Possible root causes
- Remediation options
- Post-fix recovery
This can take more than 15 minutes, resulting in viewer frustration, drop-off, and potential revenue loss.
Our vision
Don't just detect the incident. Understand it, fix it safely, and prove that it is fixed.
That vision became MediaOps CoPilot — an autonomous first responder for live media infrastructure.
2. What We Built
MediaOps CoPilot is an autonomous, multi-agent incident-response platform for live streaming.
It follows a complete closed-loop lifecycle:
Detect → Diagnose → Decide → Gate → Execute → Verify → Learn
For our hackathon demo, we focused on a realistic encoder failure affecting a live stream. The live demo covers three fault types (encoder failure, encoder overload, network degradation) and four bounded actions: restart encoder, reduce profile, switch source and failover.
3. How It Works
① Detect
- A deterministic monitoring layer continuously monitors infrastructure telemetry.
- It looks for sustained anomalies, rather than reacting to temporary spikes.
- This helps prevent false incidents.
② Diagnose
Two specialised AI agents investigate the incident in parallel:
Vision Agent
- Analyzes the actual video.
- Identifies viewer-visible issues such as:
- Freezing
- Black frames
- Macroblocking
- Video degradation
Infra Agent
- Queries infrastructure telemetry through the Grafana MCP server, such as:
- CPU
- Frame rate
- Packet loss
- Encoder health
Together they answer two critical questions:
What does the viewer see? What does the infrastructure say?
- The incident is confirmed only when the evidence corroborates the failure.
③ Decide
A Remediation Agent combines:
- Diagnostic findings
- Current incident context
- Similar previously resolved incidents from the Knowledge Base
It recommends one action from a tested set of safe remediation actions.
Important: The AI can recommend an action, but cannot execute it directly.
④ Gate — Safe Autonomy
The Safety Gate is a deterministic, fail-closed control layer.
Before anything is executed, it checks:
- Is the action approved?
- Is the target component correct?
- Is the action within the allowed remediation set?
If anything is invalid or uncertain:
BLOCK.
The gate runs eight checks before any action: allow list, incident match, action-to-fault compatibility, approved target, evidence confidence (0.8 minimum), blast radius, action budget (2 per incident) and cooldown. Any failed check blocks the action.
This creates a clear separation:
AI = Judgment Deterministic code = Authority
⑤ Execute
- Approved actions are sent to the Control Plane.
- Remediation is executed idempotently.
- Duplicate incidents or events cannot accidentally trigger duplicate fixes.
⑥ Verify
We don't assume that a successful command means successful recovery.
MediaOps CoPilot independently verifies recovery using:
- Infrastructure telemetry
- Actual video quality
The incident is closed only after sustained recovery is confirmed.
“It ran” is not the same as “it recovered.”
⑦ Learn
- The incident, evidence, remediation, and verified outcome are stored as operational memory.
- Similar future incidents can retrieve this knowledge and benefit from previous resolutions.
This creates a continuous:
Incident → Resolution → Learning → Better Future Response
loop.
4. Architecture & Technology
We built MediaOps CoPilot as an multi-agent system using Google Cloud and Google ADK.
Core technologies
- Google ADK — multi-agent orchestration and workflow management
- Gemini 2.5 Flash / Vertex AI — multimodal reasoning
- Cloud Run — hosted application deployment
- Firestore — incident state, audit trail, and operational memory
- Vertex AI Embeddings — semantic retrieval of similar incidents
- Grafana + Prometheus — monitoring and observability. The Infra Agent reads telemetry through the official Grafana MCP server (grafana/mcp-grafana, read-only; local run, the hosted demo uses the Prometheus fallback), so its diagnosis is built on live Grafana queries.
- FFmpeg — real video workload and fault simulation
- Safety Gate — deterministic authorization
- Control Plane — guarded remediation execution
- Slack — operational notifications
- Testing — 90 automated tests cover the safety gate, orchestrator, control plane, verification, reporting and knowledge-base write-back
5. Our Key Innovation
The most important architectural decision was:
AI supplies judgment. Deterministic code verifies and supplies authority.
Instead of allowing an LLM to directly control infrastructure:
AI Agents → Investigate → Reason → Recommend
Safety Gate → Validate → Authorize / Block
Control Plane → Execute
Verification → Prove recovery
This gives us agentic intelligence with controlled execution.
6. What We Learned
- Think beyond one incident: Identify recurring problems and their scale and impact across the media and
entertainment industry. - Build for scale and reuse: Turn a specific solution into a reusable capability that can benefit many.
- Agentic AI is about the workflow — The value lies in orchestration, guardrails, execution, verification, and learning.
- A powerful LLM alone is not enough to create an autonomous system.
- The real engineering challenge is designing: *Architecture decisions and trade-offs — multi-agent vs. single-agent, LLM selection, cost vs. quality vs. latency, and Responsible AI safeguards.
- Workflow orchestration
- State management
- Guardrails
- Failure handling
- Verification
- Learning loops
Different evidence needs different agents
- Video and infrastructure telemetry provide different perspectives.
- Specialized agents can investigate each evidence source independently.
- Corroborating their findings creates stronger incident confidence.
AI should recommend — not automatically have authority
- We learned to treat AI output as a proposal, not an instruction.
- This allows us to use agentic reasoning while keeping execution deterministic and controlled.
Recovery must be proven
- A successful API response or restart command does not necessarily mean the viewer experience has recovered.
- True recovery requires independent verification.
7. Key Challenges
Building a system that is both autonomous and predictable was our biggest challenge.
We had to solve for:
- Sustained anomaly detection vs. transient spikes
- Duplicate incident prevention
- Duplicate remediation prevention
- Parallel agent execution
- Workflow state management
- Conflicting evidence between agents
- Restricted and approved remediation actions
- Fail-closed safety controls
- Idempotent execution
- Auditable incident history
- Reusable operational memory
Hackathon trade-off
Rather than attempting to automate every possible media-operations scenario, we focused on one complete, realistic incident lifecycle end to end.
That allowed us to demonstrate the full loop:
Failure → Detection → Diagnosis → Decision → Safe Remediation → Verified Recovery → Learning
Current limits: the stream and telemetry come from a simulator, the demo runs a single encoder, and the hosted version reads Prometheus directly.
8. Impact
Our goal was to demonstrate measurable improvement over traditional manual incident response.
Results from our testing
- ⚡ 44 seconds average autonomous recovery (average of 5 test runs)
- ⏱️ more than 15 minutes typical manual response
- ⏱️ What this means for viewers (illustrative example): on a live stream with 50,000 concurrent viewers, a 15-minute manual response means about 750,000 viewer-minutes of broken video. At 44 seconds, that drops to about 37,000, a reduction of roughly 95%.
- 📹 Recovery verified using actual video + infrastructure telemetry
- 🧾 Every incident creates an auditable record
- 🧠 Resolved incidents become reusable operational knowledge
The result is not simply an AI chatbot.
It is a closed-loop autonomous incident-response system that can:
Investigate → Decide → Safely Remediate → Verify → Learn
9. What's Next
We started with live streaming because the impact is highly visible:
When something breaks behind the stream, every second matters.
But the architecture can extend beyond a single encoder failure or streaming scenario.
Our broader vision is an autonomous operational layer for media and entertainment that can:
Detect → Understand → Safely Act → Verify → Learn
Make It Scalable
Build MediaOps CoPilot as an API-first, multi-tenant SaaS platform that media companies can easily integrate with their existing monitoring, video, and infrastructure systems.
Tenant → API Gateway → MediaOps Orchestrator → Agents → Safety Gate → Control Plane → Verification
Scalability comes through:
- Multi-tenancy — isolated data, policies, configurations, and knowledge per company
- API-first integration — plug into existing monitoring and infrastructure without replacing them
- Reusable agents and workflows — extend the same capability to new incident types
- Event-driven architecture — scale incident processing independently as volume grows
- Configurable safeguards and remediation — adapt policies to each company's environment
This enables MediaOps CoPilot to evolve from a hackathon solution for one incident into a reusable, scalable operational capability across the media and entertainment industry.
Reactive incident management → Proactive, trusted autonomy
10. Additional Demo Video:
Simulation Encoder Overload: https://www.youtube.com/watch?v=WoWEl66Gbhc **
Our Guiding Principle
When something goes wrong behind the stream, the viewer shouldn't have to know.
MediaOps CoPilot Detect it. Understand it. Fix it. Prove it's fixed.
