Roach Watch — Agentic Incident Response & Memory Engine

Roach Watch: Agentic incident responspowered by CockroachDB persistent memory for retrieval, reasoning, fix validation, recurrence detection
DASHBOARD

Inspiration

Production incidents rarely happen in isolation. Engineers often have to investigate failures under pressure while relying on fragmented logs, documentation, and memories of previous incidents. Even when a similar incident has happened before, that knowledge can be difficult to reuse.

We wanted to build an agent where memory was not just a vector store attached to a chatbot, but the operational core of incident response.

That led to Roach Watch — an agentic incident-response system that remembers previous incidents, retrieves relevant operational knowledge, and uses that memory to make better decisions during future incidents.

Every incident should make the next incident easier to solve.

What it does

Roach Watch uses CockroachDB as its persistent memory engine, storing both structured incident data and embeddings.

The system:

Generates incident embeddings with NVIDIA NIM
Stores incident state and embeddings in CockroachDB
Uses Distributed Vector Indexing for historical incident retrieval
Combines semantic similarity with service and severity relevance
Uses NVIDIA reranking to improve retrieved results
Uses Groq for incident reasoning and recommendations
Exports postmortems to AWS S3
Uses Recurrence Watch to detect when previous fixes fail or incidents return
Updates trust in historical fixes using operational evidence
Uses the CockroachDB Managed MCP Server for agent/database interaction
Includes a deterministic evaluation suite to measure retrieval quality

The core loop is:

```text id="h7h8b4" Incident → Store → Retrieve → Reason → Act → Validate → Update Memory


## How we built it

Our stack includes:

* **CockroachDB Cloud** — transactional data, persistent memory, and vector search
* **CockroachDB Distributed Vector Indexing** — semantic retrieval
* **CockroachDB Managed MCP Server** — agent/database interaction
* **NVIDIA NIM** — embeddings and reranking
* **Groq** — reasoning
* **AWS S3** — postmortem storage
* **Node.js / Express** — backend
* **React / TypeScript** — frontend
* **Vercel + Render** — deployment

A key architectural decision was keeping structured incident state and vector memory in CockroachDB rather than maintaining a separate vector database. This keeps the application's source of truth and agent memory together.

## Challenges we ran into

Our biggest challenge was making memory **useful and reliable**, rather than simply storing data.

We encountered a retrieval-threshold bug that silently filtered out relevant incidents. Because the application didn't crash, we found it by examining raw retrieval results and comparing test runs.

We also faced infrastructure limitations around production-level remediation validation. Instead of simulating capabilities we could not prove, we built narrower tests around behavior we could actually verify.

These challenges taught us to validate system behavior with real evidence rather than assuming that successful execution meant correct results.

## Accomplishments that we're proud of

We're proud that Roach Watch goes beyond basic RAG.

* **Persistent operational memory** using CockroachDB
* **Hybrid retrieval + reranking** for more relevant historical incidents
* **Recurrence-aware fix memory** that can reduce trust when previous fixes fail
* **Managed MCP integration** for agent access to its memory infrastructure
* **Deterministic evaluation** instead of relying on the LLM to evaluate itself
* **Real AWS S3 integration** for postmortem storage

Most importantly, the agent doesn't just retrieve memory—it **uses that memory during reasoning and updates it based on subsequent outcomes**.

## What we learned

We learned that an effective agent is more than an LLM. It needs persistent state, retrieval, tools, evaluation, and reliable infrastructure.

We also learned that vector search alone isn't enough to create useful memory. Memory becomes valuable when the system can:

**remember → retrieve → reason → act → validate → update.**

The project also reinforced the importance of testing for silent failures, especially in retrieval systems where incorrect results may not produce obvious errors.

## What's next for Roach Watch — Agentic Incident Response & Memory Engine

Next, we want to connect Roach Watch to real-time production telemetry, logs, and metrics so incidents can be detected and enriched automatically.

We also want to strengthen remediation validation using real infrastructure signals, expand memory to include runbooks and service dependencies, and introduce controlled autonomous remediation with human approval.

Our long-term goal is to build an incident-response agent whose **operational memory becomes more useful with every verified incident**.