EvidenceOps

• 

• 

Inspiration

Infrastructure incidents rarely fail cleanly. A cache runs low on memory, a mutex gets confused, or a WordPress core CVE is disclosed across a portfolio of managed sites. The responder’s real bottleneck is often context: what does this asset affect, has something comparable happened before, and which parts of that past resolution still apply?

EvidenceOps is built around that problem: CockroachDB holds durable infrastructure context and incident memory, AWS Bedrock assists with evidence-gated investigation, and remediation remains outside the system until a person makes an explicit decision.

What it does

EvidenceOps turns an infrastructure signal into a reviewable incident investigation.

It retrieves infrastructure context through a constrained CockroachDB Managed MCP read path and searches resolved incident memory using CockroachDB vector similarity rather than keyword matching. It presents a match delta: what is shared with a historical incident, what has changed, and what must not be assumed to transfer.

An optional AWS Bedrock investigation works from a server-constructed, read-only evidence bundle. The model must cite supplied evidence IDs; if the response fails validation, the investigation is rejected for the operator to rerun.

The resulting investigation separates Investigator, Historian, Planner, and Reviewer findings into structured, inspectable sections. No model tool can write data, execute a command, or approve a change. EvidenceOps also implements Slack Events API intake. Once configured, signed and allowlisted Slack messages are redacted, deduplicated, filtered for bot/self messages, and stored as pending human review. A reviewed intake may be explicitly promoted into a needs_review incident; it never becomes an incident automatically.

Every recorded decision and approved investigation is available through the CockroachDB-backed audit replay. The current build has no remediation write-back capability.

How we built it

CockroachDB is the backbone for deployed infrastructure context, incident memory, audit data, encrypted Slack settings, intake deduplication, and review records.

Incident records combine structured operational fields with vector embeddings. A Distributed Vector Index, incidents_embedding_idx, supports nearest-neighbour retrieval, and the repository includes an EXPLAIN verification script to confirm that the intended vector query uses that index on the deployed cluster.

The Managed MCP integration is intentionally narrow. In strict MCP mode, infrastructure-context lookup fails closed if the Managed MCP configuration or request fails; it does not silently switch to SQL. The application exposes only one fixed select_query catalog join after validating the incident ID and database identifier. It does not expose a generic MCP client to the browser or model.

Two further read-only CockroachDB evidence sources are added before an AI investigation:

• A bounded CockroachDB Cloud API request for cluster-health context.
• A fixed SHOW STATISTICS FOR TABLE defaultdb.public.incidents adapter, traced to a pinned version of CockroachDB’s auditing-table-statistics Agent Skill. It is not a general skill runner or SQL executor.

AWS Bedrock provides the optional investigation. The model has one available tool, read_incident_evidence, which returns the complete read-only evidence bundle. The output must follow a structured contract and cite valid evidence IDs before it is saved.

Slack intake uses HMAC request verification, a five-minute replay window, an explicit channel allowlist, durable event-ID deduplication, redacted logs, and bot/self-message filtering. Accepted messages enter a review queue; only an operator can review, dismiss, or promote them.

The application is a TypeScript/React app built with Vinext. It is deployed on AWS Lightsail with Docker Compose bound to loopback, behind Nginx and TLS. The dashboard and protected APIs use signed, HTTP-only local operator sessions; the landing page, sign-in route, and health endpoint remain public.

Challenges we ran into

The vector index needed verification beyond schema creation. We added an EXPLAIN check so the project could demonstrate that the intended nearest-neighbour query uses incidents_embedding_idx, rather than merely claiming that an index exists.

Keeping the Managed MCP boundary meaningful also required care. Strict MCP mode fails loudly for the infrastructure-context route instead of silently falling back, while the application itself remains limited to a fixed read-only catalog query.

The subtler Slack risk was recursive intake. If bot-originated events were not filtered, a future acknowledgement or bot message could be processed as another incoming signal. The implementation treats this as a correctness boundary and tests it directly.

Accomplishments that we’re proud of

• CockroachDB vector-backed historical memory with a verification path for index use.
• A constrained Managed MCP integration that exposes one fixed, read-only catalog query.
• An evidence-gated Bedrock investigation that rejects invalid evidence citations.
• Slack intake that remains a redacted, human-reviewed draft until explicit promotion.
• CockroachDB-backed decision and investigation audit replay.
• An anonymized seed portfolio combining operationally derived cases with clearly labelled synthetic sibling and demo records.

What we learned

The valuable part of an incident-response agent is not autonomous action. It is the disciplined handoff between grounded context, remembered experience, model-assisted reasoning, and a person who remains accountable for what happens next.

Historical incidents are most useful when the system can show not only why they are similar, but also exactly what is different and what cannot safely be transferred.

What’s next for EvidenceOps

More anonymized operational seed incidents; additional monitoring-source integrations; multi-user organization membership and authorization; and production third-party authentication to replace the current local operator account.

Built With

• aws-bedrock
• aws-lightsail
• cockroachdb
• letsencrypt
• nginx
• react
• slack-api
• typescript
• zod