-
-
Logo
Inspiration
Production incidents are expensive because finding the fix still takes too long.
When something breaks in production, most teams run the same manual loop. An alert fires, someone wakes up, opens dashboards, searches logs, checks recent deploys, figures out who owns the service, and then decides whether to rollback.
That process can take 20 to 40 minutes. It pulls senior engineers out of deep work, or out of bed. And a lot of the time, the root cause is something that could have been identified automatically.
The standard answer to this problem has been better alerting. More dashboards, more Slack notifications, more runbooks. But that does not really solve the problem.
The problem is that turning that information into a decision still requires a human to manually investigate everything, every single time.
We built VoiceOps because we think the investigation part of incident response should be automated. The human should be making the call on what to do rather than spending 30 minutes trying to find the cause.
VoiceOps is your AI on-call engineer when prod breaks.
What it does
VoiceOps is an AI incident responder for production outages.
It detects production issues, traces the likely cause to the faulty deployment, calls the on-call engineer by phone, waits for approval, and rolls back the bad commit after the engineer signs off.
When Dynatrace Davis AI flags a problem, VoiceOps starts a two-phase response automatically.
Phase 1: Triage and mitigation
- Queries Dynatrace through the MCP server to get full incident context and affected entities
- Pulls recent GitHub commits and compares them against the incident window
- Identifies the most likely suspect commit and assigns a confidence score
- Places a real outbound phone call through VAPI to brief the on-call engineer in plain English
- Explains the incident, affected service, suspect commit, confidence level, and recommended action
- Creates an approval request so the engineer can approve by voice or keypad
- Triggers a GitHub Actions rollback workflow to the last known-good commit through the GitHub REST API
- Polls the rollback workflow until it completes
- Re-queries Dynatrace to verify that the problem is resolved
- Sends live voice updates to the engineer at each milestone
Every rollback requires explicit human approval, regardless of confidence level.
Phase 2: Post-incident analysis
After the service is restored, VoiceOps runs the deeper investigation.
- Runs DQL log queries and anomaly detection to confirm the root cause
- Calculates blast radius, failed request count, and error trends
- Sends a final voice update with the RCA summary
- Files a structured post-incident report as a GitHub issue
- Closes the tracking issue
The result is simple.
Instead of waking up to a vague alert and manually digging through dashboards, the engineer gets a phone call that says what broke, what likely caused it, what VoiceOps recommends, and what needs approval.
The engineer still owns the decision but VoiceOps does the investigation and execution.
How we built it
Agent layer
The core agent runs on Google ADK with Gemini 3.5 Flash as the reasoning engine through Vertex AI.
The agent has 18 tools across triage, mitigation, rollback, verification, and post-incident reporting. It follows a structured incident response workflow defined in its system prompt, and the ADK web runtime handles sessions and exposes a live event stream for monitoring.
Observability
Dynatrace MCP Gateway is the agent's main observability interface.
It gives VoiceOps access to Davis AI problem detection, DQL log queries, entity resolution, anomaly detection, and Dynatrace documentation. This was one of the biggest unlocks for the project because it let the agent reason over real observability data without us building a full custom observability integration from scratch.
Voice layer
VAPI handles all outbound phone calls.
VoiceOps injects incident context into the call so the engineer gets a clear briefing immediately. Then, while the rollback is happening, VoiceOps uses VAPI's live call control API to push real-time status updates into the same call.
So the engineer does not have to open a dashboard to know what is happening. They hear the progress live.
Human approval
A FastAPI approval server handles the VAPI webhook, resolves approvals by voice or keypad, and exposes a browser dashboard as a fallback.
The agent creates an approval request, places the call, and then polls the approval server for up to five minutes. If the engineer does not approve, VoiceOps stands down and escalates instead of taking action on its own.
Rollback
VoiceOps triggers a workflow_dispatch event on a GitHub Actions workflow through the GitHub REST API.
The workflow finds the last safe commit, builds a new Docker image, pushes it to Artifact Registry, and deploys it to Cloud Run.
Infrastructure
The ADK agent, approval server, and demo target service all run on Google Cloud Run.
Cloud Build handles container builds. The demo target service has runtime break and fix endpoints, and the approval server has a single /demo/start endpoint that sets up the full demo in one command.
That means a judge can trigger the full end-to-end run without touching GCP, changing environment variables, or manually breaking the service.
Challenges we ran into
Async human approval
The hardest part was making the agent wait for human approval without blocking the whole system.
The agent needs to place a phone call, wait for the engineer to respond, and then continue the rollback only if approval is granted. ADK does not have a native "wait for webhook" primitive, so we built an approval server that sits between VAPI and the agent.
The VAPI webhook resolves the approval asynchronously, and the agent polls a /pending endpoint with a five-minute timeout.
Live phone call updates
We wanted the engineer to stay on the call and hear live updates as the rollback progressed. That meant using VAPI's live call control API to inject status updates mid-call.
Getting the timing right took a few iterations, especially because the updates needed to feel natural while still being tied to real rollback milestones.
Gemini rate limits
The agent makes several sequential MCP calls during triage. Under load, that can trigger rate limit errors.
We added exponential retry using HttpRetryOptions with a 15-second initial delay and three attempts, so a temporary 429 does not fail the entire incident response flow.
Reproducible judging demo
A real demo requires a few moving pieces: a live service, a simulated production break, Dynatrace detection, a phone call, approval, rollback, and verification.
We did not want judges to have to manually set that up.
So we added runtime break and fix endpoints to the target service, plus a /demo/start endpoint on the approval server. That endpoint handles phone override, service break, and demo mode in one command, so the full run can be triggered with a single curl request.
Accomplishments that we're proud of
It is a real closed loop
VoiceOps detects the issue, investigates the cause, calls the engineer, gets approval, rolls back the bad commit, verifies recovery, runs post-incident analysis, and files the report.
The only required human step is approval.
The voice interface actually changes the experience
Dashboards assume the engineer is already at their laptop.
But during on-call, the engineer might be asleep, away from their desk, or not ready to context switch into five different tools.
A phone call meets them where they are. VoiceOps explains the issue in plain English, asks for approval, and keeps them updated while the rollback happens.
That is a very different experience from another Slack alert.
Human approval is built into the system
We chose not to auto-rollback, even when the confidence score is high.
Production rollbacks have consequences. The agent can investigate, recommend, execute, and verify, but a human should still own the decision to change production.
The demo is easy to run
A judge with no familiarity with the codebase can trigger a full end-to-end production incident demo with one command.
What we learned
Dynatrace MCP is a serious force multiplier
Dynatrace MCP gave the agent access to problem detection, DQL, anomaly detection, entity context, and documentation without us spending the whole hackathon building custom observability plumbing.
The biggest lesson was that MCP makes observability data feel like a first-class part of the agent's reasoning context.
Instead of the agent guessing from a static prompt, it can ask the observability system what is happening right now.
Voice is underrated for operational AI
Most AI developer tools are built for users that are already sitting in front of a screen.
A phone call is immediate. It can wake the right person up, explain the situation quickly, and get a decision in less than a minute.
Voice also forces the agent to be clear. If the agent cannot explain the incident well enough for a human to approve the action, it should not be taking that action.
What's next for VoiceOps
- On-call routing through PagerDuty or OpsGenie, so calls go to the right engineer based on the current rotation
- Multi-service blast radius detection for incidents that affect more than one service
- Configurable approval SLAs, including escalation to a secondary contact if the primary does not respond
- Slack and Teams fallback channels for teams that want async approval
- Confidence-tunable autonomy, so each team can decide what the agent can do automatically and what always needs approval
- Incident memory across past failures, runbooks, ownership, and previous fixes
- Safer remediation options beyond rollback, like restarting a service, scaling a deployment, or disabling a feature flag
Log in or sign up for Devpost to join the conversation.