Sentinel

An SRE agent with lifelong memory in CockroachDB. It recalls past incidents, avoids mitigations that failed before, and gets smarter with every outage it resolves.

Comment

Inspiration

Most incident-response agents are stateless — they re-diagnose the same outage from scratch every time and happily repeat mitigations that already failed last week. We wanted to build an agent where memory isn't a bolt-on feature but the actual product: an SRE agent that remembers every incident it has ever handled, in a database that treats transactional state and semantic memory as one thing instead of two systems glued together with ETL.

What it does

Sentinel watches for infrastructure alerts. When one fires, it semantically recalls similar past incidents from a CockroachDB vector index, reasons over live telemetry, applies exactly one mitigation at a time, verifies recovery, and — on resolution — writes a new postmortem back into the same memory it just read from. The next similar incident recalls that postmortem too, including which mitigation worked and which one explicitly didn't. Before any mitigation that could touch state, it also checks its own database cluster's health via CockroachDB's ccloud CLI — a triage agent inspecting its own infrastructure before acting on someone else's.

How we built it

The backend is .NET 10: an agent loop that recalls memory, reasons with tool use, executes tools, and audits every step as an immutable row in CockroachDB. The schema holds four kinds of memory in one database — incident state, an append-only event log, an immutable action audit trail, and vector-indexed postmortems/runbooks — all written in the same transactions, so "what happened" and "what it means" can never drift apart. A React dashboard replays the agent's reasoning live, including similarity meters for every memory it recalls. Resolved postmortems are archived to S3 as well as embedded back into CockroachDB.

Challenges we ran into

The big one: a brand-new AWS account. Mid-hackathon we hit an account-wide Amazon Bedrock on-demand token cap — confirmed directly in Bedrock's own Playground, independent of any of our code, across multiple models including Amazon's own Nova. AWS support couldn't clear it inside the hackathon window. Rather than block the whole submission on a support queue, we moved the reasoning and embedding calls to direct APIs — Anthropic's Messages API for Claude, Voyage AI for embeddings — while keeping CockroachDB, S3, and ccloud exactly as designed. The hackathon rules explicitly allow any model combination, so this is a supported configuration, not a workaround. We also hit a real correctness bug along the way: Claude Sonnet 4.5 calls multiple tools in parallel by default, and our first implementation only handled one tool call per turn — fixed by collecting every tool call in a turn and returning all results together, as Anthropic's API requires.

Accomplishments that we're proud of

A fully working end-to-end agent loop: real memory recall with similarity scoring, real parallel tool use, real mitigation decisions informed by what failed before, real resolution, real postmortem writeback — all persisted in one CockroachDB cluster. And a genuine recovery story: swapping out a broken cloud dependency mid-project without touching the memory layer, the tool-use loop, or the audit trail.

What we learned

That "the agent trusts its memory" only means something if the memory actually encodes failure, not just success — our seed postmortems explicitly record mitigations that didn't work, and the system prompt tells the agent to respect that. We also learned, the hard way, that new cloud accounts come with invisible guardrails that have nothing to do with your code or your credit balance — and that the right response is to swap the dependency, not wait on a support queue.

What's next for Sentinel

Wiring the Lambda alert ingester to a real CloudWatch alarm for a fully unattended demo, swapping the deterministic demo telemetry for a live CloudWatch/Prometheus client, and adding CockroachDB's Managed MCP Server at agent runtime (currently used only for dev-time schema inspection) for ad hoc operator queries alongside the agent's automated loop.

Built With

Share this project:

Updates

Submission history