IncidentMind AI

Inspiration

AI can suggest a convincing solution to a software incident, but that does not mean the solution actually worked. Reusing an incorrect diagnosis can make future incidents worse.

That inspired IncidentMind AI, built around one principle: remember what was verified, not merely what an AI suggested.

What it does

IncidentMind investigates software failures using observable evidence, ranks possible root causes, verifies whether recovery actually succeeds, and only then promotes the resolution into trusted memory.

In our demo, a checkout service experiences database connection pool exhaustion. IncidentMind observes the saturated pool and failed checkout probe, identifies the likely root cause, and initially marks it only as SUPPORTED.

After recovery, fresh measurements confirm that the service is healthy again. Only then does the resolution become VERIFIED. When a similar incident occurs again, IncidentMind retrieves and reuses the previously verified memory instead of starting from zero.

Evidence → Verify → Remember → Reuse

How we built it

The application uses FastAPI for the backend and React + TypeScript for the reliability dashboard.

CockroachDB Cloud provides the persistent agentic memory layer. Incident memories use VECTOR(32) embeddings and vector similarity search to retrieve relevant previous incidents. A trust-scoring layer prioritizes verified memories and penalizes rejected or unverified knowledge.

We also integrated CockroachDB Managed MCP as an OAuth-authorized, read-only inspection path.

Amazon S3 is used for securely storing sanitized investigation artifacts with server-side encryption.

We additionally built a deterministic demo environment and a real local failure lab so the complete investigation, recovery, verification, and memory-reuse workflow can be demonstrated safely.

Challenges we ran into

The biggest challenge was preventing an AI-generated diagnosis from automatically becoming trusted knowledge. Confidence alone is not proof, so we designed a separate verification stage based on fresh post-recovery evidence.

Another challenge was creating a realistic but safe software failure. We built a checkout service that intentionally exhausts its database connection pool, allowing IncidentMind to investigate actual runtime measurements without affecting a production system.

We also separated public demo functionality from private cloud credentials so the project remains safely demonstrable without exposing secrets.

Accomplishments that we're proud of

We built an end-to-end reliability workflow where operational memory is earned through verification rather than simply generated by AI.

IncidentMind can reproduce a real database pool exhaustion failure, investigate observable evidence, rank hypotheses, verify recovery, store the proven resolution, and reuse that verified experience during a subsequent incident.

We also successfully combined CockroachDB persistent vector memory, Managed MCP, AWS S3, FastAPI, React, auditability, and trust-aware memory ranking into one working system.

What we learned

We learned that useful agentic memory is not simply about storing more information. In reliability engineering, what an agent is allowed to remember matters as much as what it can retrieve.

We also learned how transactional data, vector similarity search, verification signals, and cloud artifacts can work together to create auditable operational memory.

What's next for IncidentMind AI

Next, IncidentMind can connect to production observability sources such as metrics, logs, traces, deployment events, and alerting platforms.

The long-term goal is an autonomous reliability agent that continuously learns from real incidents while maintaining a clear distinction between what was suggested, what was rejected, and what was actually proven to work.

Built With

• amazon-web-services
• cockroachdb
• fastapi
• managed
• mcp
• python
• react
• s3
• typescript