Incident Triage Agent

Agent streaming 6-step triage workflow in real time Investigate, Diagnose, Present Findings, Runbook, Email, Log
MongoDB Atlas historical match detected
Agent streaming 6-step triage workflow in real time

Inspiration

Every engineering team I've worked with has the same problem: when an incident hits at 2am, the on-call engineer spends the first 20 minutes figuring out if anyone has seen this before. That tribal knowledge lives in Slack threads, runbook wikis nobody updates, and the memories of engineers who might not even be at the company anymore.

MongoDB Atlas is a perfect fit for organizational memory it's flexible enough to store incident records of any shape, and fast enough to query under pressure. I wanted to build something that makes that knowledge instantly accessible at the moment it's most needed.

What it does

Incident Triage Agent is an AI-powered ops tool that investigates infrastructure alerts end-to-end:

Investigate — analyzes the incoming alert and searches MongoDB Atlas for similar historical incidents
Diagnose — identifies the most likely root cause with a confidence level, cross-referencing past patterns
Present Findings — summarizes what happened, what's affected, and what worked before
Generate Runbook — creates a step-by-step remediation plan referencing successful past resolutions (human approval gate)
Draft Email — writes a professional stakeholder communication ready to send (human approval gate)
Log Incident — saves the complete record back to MongoDB Atlas so every future engineer benefits

The result: a new engineer hits the same incident and finds the diagnosis, runbook, and resolution in seconds instead of hours.

How I built it

Google Cloud ADK 2.1 as the agent framework, with gemini-3.1-pro-preview as the reasoning engine
MongoDB Atlas as the persistent memory layer, accessed via a hosted MCP server on Google Cloud Run
MCP (Model Context Protocol) to give the agent native read/write access to Atlas — querying historical incidents in Step 1 and logging new ones in Step 6
Google Search Tool and URL Context Tool as sub-agents for external research when needed
FastAPI + vanilla JS for the dashboard — a dark ops-themed 3-panel interface with live SSE streaming, step-by-step reasoning cards, and a real-time MongoDB history panel

Challenges I ran into

ADK agent naming: ADK 2.x requires agent names to be valid Python identifiers, but the project folder had hyphens. Solved by moving the agent into a properly-named package subfolder.
CORS with file:// URLs: Opening the dashboard as a local file blocks cross-origin requests. Solved by serving everything through a single FastAPI server that wraps the ADK app.
MCP response parsing: The MongoDB MCP server wraps document content in security-tagged blocks. Had to write a JSON scanner that finds the first valid array in the response rather than relying on fixed tag positions.
Agent output routing: Gemini tends to output all steps in one response. Built a step-boundary detector that parses STEP N markers in the stream and routes content to the correct card in real time.

Accomplishments that I'm proud of

The MongoDB right panel populates instantly on page load with live Atlas data no mocking, no fixtures
The human approval gates feel natural the agent pauses and waits, it doesn't just steamroll through
Every incident logged during a demo becomes a real historical record that future triages can match against
The dashboard actually looks like something an ops team would use

What I learned

MongoDB Atlas as an agent memory layer is genuinely powerful. The MCP server abstraction means the agent can query and write to Atlas using natural language tool calls no custom database code needed. The combination of ADK's multi-agent architecture and MongoDB's flexible schema makes it easy to store incidents of any shape and retrieve them by similarity.