login page
Dashboard 0
Dashboard 3
Dashboard 1

IncidentMind

Inspiration

Shashwath had spent a week on-call watching a team re-diagnose an incident that had basically happened before — same service, similar symptoms, same eventual fix — except nobody remembered that until someone dug through old Slack threads an hour in. That hour is the part that stuck with us. The knowledge wasn't missing; it was just locked in someone's memory or buried in a channel nobody thought to search.

We kept coming back to the same question: what if the system remembered so the team didn't have to? Not another dashboard, not another runbook wiki — something that actually reasoned about the incident in front of you using everything similar that had happened before. That's what became IncidentMind.

What it does

IncidentMind takes an incident description and technical context and turns it into a fast, actionable analysis. It:

Summarizes the incident so anyone jumping in — even mid-outage — can understand what's happening at a glance.
Identifies likely root causes, reasoned from the incident details and grounded in similar incidents from the past.
Recommends concrete remediation steps, not just a diagnosis.
Remembers. Every incident it processes becomes part of its memory, so the next similar incident gets analyzed faster and more accurately — turning tribal knowledge that used to live in people's heads into something the whole team can draw on.

Under the hood, a safety/validation layer checks AI-generated recommendations before they're surfaced, so the system supports a human's judgment during an incident rather than replacing it.

How we built it

We started small: could we get a model to take a raw incident description and produce a reasonable root-cause guess? That worked well enough to be encouraging and badly enough to be humbling — early outputs were plausible-sounding but not grounded in anything, which is exactly the kind of confident-wrong answer you don't want during an incident.

So the architecture grew from there. Instead of one model doing everything, we split the work into specialized agents — one to understand and structure the incident, one to reason about likely causes, one to rank those causes against historical precedent, one to turn a diagnosis into concrete remediation steps, and a safety/validation layer to catch recommendations that shouldn't be surfaced without a human sanity check. An orchestrator ties these together.

CockroachDB sits underneath as the system of record for incidents and memory — we needed something that wouldn't fall over during the exact conditions (load, partial outages) an incident-response tool is most likely to be used in. AWS handles the AI/compute side, including model access through Bedrock.

Challenges we ran into

Getting foundation model access working through Amazon Bedrock took longer than we expected — provider-specific approval steps for Anthropic models weren't something we'd budgeted real time for, and it briefly stalled the whole AI layer while we sorted out access.

Keeping memory retrieval reliable was harder than it looked on paper, too. It's one thing to store past incidents; it's another to reliably surface the relevant ones when a new incident comes in with slightly different wording or symptoms. We also had to plan for the AI layer being unavailable or slow — an incident tool that goes silent exactly when a real incident hits is worse than useless — so we built clear service boundaries rather than wiring the whole platform tightly to one model call.

Accomplishments that we're proud of

Getting a multi-agent system to reason about an incident in coherent stages — understanding, ranking, recommending — instead of one model trying to do everything in a single pass.
Building persistent memory that actually improves the system's usefulness over time, rather than just logging history nobody looks at again.
Designing the AI layer with real service boundaries, so the platform degrades gracefully instead of breaking outright when an external AI dependency is slow or unavailable — which matters a lot for a tool meant to be trusted during outages.
Getting Bedrock and Anthropic model access fully working end-to-end despite the setup friction, and using that to power the reasoning and recommendation agents.

What we learned

The biggest lesson wasn't technical, exactly — it was that a good answer isn't the same as a useful one. Early versions could summarize an incident accurately and still leave someone unsure what to actually do next. Getting from "accurate summary" to "here's your next action" meant thinking hard about memory retrieval (pulling the right past incidents, not just similar-sounding ones), about reasoning steps the model needed to show its work on, and about where a human needed to stay in the loop rather than trusting the output blindly.

We also learned a lot about the mechanics of running specialized agents against a shared memory store — how easily one agent's assumptions can quietly poison another's output if the handoffs aren't well-defined, and how much better the system got once we added explicit validation between stages instead of trusting each agent's output at face value.

What's next for IncidentMind

Automatic ingestion — pulling incidents directly from tools like PagerDuty, Slack, or log pipelines instead of relying on manually pasted context.
Feedback loops — letting engineers rate or correct recommendations so the memory system gets more accurate over time, not just larger.
Deeper integrations — surfacing recommendations directly inside the tools teams already use during an incident, rather than a separate destination to check.
Expanded safety validation — more rigorous checks on higher-risk recommendations (e.g. anything involving production changes) before they're ever shown to an on-call engineer.