Inspiration

During a critical enterprise incident, teams often have to switch between monitoring dashboards, deployment logs, tickets, documentation, and communication tools just to understand what went wrong. We wanted to explore a better approach: what if an AI system could coordinate this investigation instead of making humans do all the searching and connecting?

This inspired AI Incident Commander — an agentic system designed to investigate incidents, coordinate specialist agents, recommend remediation, and verify the resolution while keeping humans in control of high-impact actions.

What We Built

We designed a multi-agent incident response workflow around:

Detect → Investigate → Correlate → Decide → Approve → Act → Verify

The Incident Commander acts as the central coordinator and can work with specialized agents such as:

  • Monitoring Agent — analyzes service health and metrics.
  • Deployment Agent — checks recent deployments and changes.
  • Knowledge Agent — searches previous incidents, documentation, and runbooks.
  • Impact Agent — identifies affected users and services.
  • Remediation Agent — handles approved recovery actions.
  • Verification Agent — confirms whether the incident has actually been resolved.

We created a clickable Figma prototype demonstrating the complete journey from incident detection to investigation, root-cause analysis, human approval, and verified resolution.

How It Works

For our prototype scenario, a CRM service experiences a sudden increase in API errors after a deployment.

The Incident Commander receives the incident and coordinates the specialist agents. The agents gather different pieces of evidence, such as monitoring data, deployment history, previous incident knowledge, and impact information.

The Commander then presents the probable root cause and a recommended remediation action.

For potentially risky actions, the system does not blindly act. Instead, it presents the recommendation to a human operator for approval.

After approval, the remediation is performed in the proposed workflow, followed by automated verification. The system then produces an incident summary containing the root cause, action taken, and resolution status.

The numbers and incident data shown in the prototype are simulated demo data created to demonstrate the workflow.

What We Learned

One of our biggest learnings was that building an agentic system is not simply about adding an LLM to an application.

The real value comes from giving agents:

  • Clear responsibilities
  • Access to relevant tools and knowledge
  • The ability to delegate tasks
  • Evidence to reason over
  • Controlled permissions
  • Clear boundaries for autonomous actions

We also learned that human-in-the-loop design is important for enterprise AI. An AI system can investigate and recommend an action, but high-impact production changes should have appropriate approval and safeguards.

Challenges

One of our main challenges was designing a complex multi-agent workflow while keeping it understandable and feasible for a hackathon build.

We also had to decide which parts should be autonomous and which should require human approval. Another challenge was representing integrations with monitoring systems, ticketing platforms, deployment systems, and knowledge bases in a way that could eventually be implemented through tools and MCP.

For the qualification round, we focused on creating a clear and realistic product experience through our interactive prototype, while keeping the architecture extensible for the 24-hour implementation phase.

What's Next

During the in-person build, we aim to turn the prototype into a working agentic system with real tool interactions, MCP-based integrations, stronger orchestration, and a functional end-to-end incident workflow.

Our goal is simple:

Don't just tell teams that something went wrong. Help them understand why, coordinate the response, and verify that it is actually fixed.

Built With

  • agentic
  • ai
  • aiops
  • automation
  • figma
  • llm
  • saas
Share this project:

Updates