Incident Commander AI
The Problem
When a production incident happens, engineers have to move quickly while making decisions that can have serious consequences. A high-error-rate deployment can require investigation, evidence gathering, remediation planning, human approval, rollback, recovery verification, and finally documenting what happened.
We wanted to build an AI system that could handle this workflow instead of simply answering questions in a chat window.
What We Built
Incident Commander AI is an autonomous incident-response agent designed to investigate and remediate production incidents while keeping humans in control of high-impact actions.
The system follows a structured workflow:
- Detect — identifies an incident and analyzes the available evidence.
- Investigate — uses an AI Detective agent to examine logs and determine the likely cause.
- Propose remediation — generates a concrete remediation proposal.
- Human approval — pauses before the destructive/high-impact action and requires explicit human approval.
- Execute remediation — performs the approved rollback.
- Verify recovery — gathers fresh evidence after remediation instead of assuming the incident is fixed.
- Resolve and document — records the resolution and generates a postmortem.
A Mission Control dashboard provides a visual interface for watching the incident progress and approving the remediation.
Why It Is Different
This is not designed as a chatbot that simply recommends commands.
The important part is the workflow and its safety gates. The agent can investigate asynchronously, maintain incident state, create remediation proposals, wait for human approval, execute the approved action, and verify the result.
High-impact operations are protected by explicit approval and verification gates. Security tooling also provides defense-in-depth around agent tool usage.
How We Built It
The project uses Google's agent ecosystem and Google Cloud infrastructure.
- Gemini for the AI reasoning and agent workflows
- Google ADK for agent construction and tool-based orchestration
- Google Cloud Run for the application runtime
- Firestore for incident state and persistence
- Pub/Sub for event-driven pipeline integration
- FastAPI for the Mission Control API
- Python for the backend and agent implementation
The system contains separate Commander, Detective, and Remediation responsibilities, with security controls around tool execution.
What We Learned
One of the biggest lessons was that autonomous agents need stronger boundaries than a normal conversational application.
We learned to separate:
- investigation from remediation
- proposals from execution
- AI decisions from human authorization
- remediation from recovery verification
- application state from infrastructure integrations
We also learned that verification must use fresh evidence. Successfully executing a rollback is not the same thing as proving that the incident is resolved.
Challenges
The most challenging part was integrating multiple agent stages while preserving the approval and verification guarantees throughout the workflow.
We also had to ensure that the Mission Control API, agent orchestration, security interceptor, and Google Cloud integrations could coexist without weakening the existing safety gates.
The final deterministic test suite contains 179 passing tests, with live Gemini-dependent tests separated from the deterministic suite.
The Result
Incident Commander AI demonstrates how an AI agent can move beyond chat and operate as a structured incident-response workflow — investigating problems, proposing actions, waiting for human authorization, performing remediation, verifying recovery, and documenting the result.
The goal is not unrestricted autonomy.
The goal is useful autonomy with deliberate human control at the moments that matter.
Log in or sign up for Devpost to join the conversation.