NineLives

Never dies

Comment

Inspiration

The most-repeated complaint among on-call engineers isn't the hours — it's that most 3am pages are for systems you didn't build and don't understand. The Google SRE book names the exact failure mode this produces: under stress, engineers grab the tempting heuristic and fall for confirmation bias, because "stress hormones impair cognitive functions and cause suboptimal decision making." The engineers who diagnose incidents fast are the ones who remember — "we saw this exact pattern last month; it looked like X but was actually Y." That hard-won memory is the real asset, and it walks out the door when people leave.

Companies are now handing on-call work to AI agents. But an agent is only as good as what it can remember — and if its memory goes down, the agent doesn't degrade, it stops. That gave me the thesis for NineLives: an on-call agent whose memory is the thing that makes it useful, built on a database whose whole reason for existing is that it never goes down.

The name is a triple pun on resilience — cockroaches are nearly impossible to kill, cats have nine lives, and uptime is measured in "nines" (99.999%).

What it does

NineLives is an autonomous on-call SRE agent. Given an incident alert, it runs a four-step loop:

  1. Reads live state — the database's running queries, locks, and cluster health.
  2. Recalls the past — semantic search over its memory of prior incidents finds similar cases and the fixes that resolved them.
  3. Acts — applies a safe, whitelisted remediation, using memory to avoid the trap that live state alone walks it into.
  4. Remembers — writes the new incident and its resolution back to memory as a durable record plus embeddings, and archives the full evidence trace.

The demo centers on a lock-contention incident engineered so that every clue in live state points at the wrong session — the longest-running query is a decoy, the lock duration matches an innocent victim, and CockroachDB's lock table deliberately never exposes which session holds the lock. Only memory identifies the real culprit.

And I can prove memory matters, not just claim it. A built-in ablation runs the identical fault with recall disabled: with memory, the agent cancels the lock holder and resolves the incident with zero collateral; without memory, it falls for the decoy, cancels the wrong query, and leaves the incident burning while destroying the day's aggregates — both at high confidence. Same fault, same live state; memory is the only variable, and it changes the outcome.

How we built it

Memory layer — CockroachDB. Each incident is one row holding both its structured fields and its embeddings, indexed with a C-SPANN distributed vector index so semantic recall runs inside the operational database — no separate vector store, no consistency gap. Every incident stores two vectors: a symptom-signature vector for alert-time retrieval, and a resolution vector for root-cause clustering.

Two read/write paths. Reads and inspection go through the CockroachDB Cloud Managed MCP Server (read-only). Writes go through a direct, strongly-consistent SERIALIZABLE transaction — the incident record, both embeddings, and the S3 artifact pointer all committed atomically.

Live state comes from CockroachDB's crdb_internal introspection. Remediation is a whitelist of safe, reversible actions — no destructive operations, ever, validated against injection and malformed input.

AWS — Amazon S3 stores a full evidence bundle (reasoning trace, tool outputs, before/after state) per resolved incident, keyed by incident ID, with the key stored on the row so the pointer resolves both directions.

Agent brain. The LLM backend is pluggable behind a clean interface; the demo runs on Google Gemini's free tier, with an Amazon Bedrock (Converse) backend also implemented. The whole system runs against a local multi-node cluster for the node-kill demonstration, and against a CockroachDB Cloud cluster for the MCP memory path.

Challenges we ran into

The hardest problem was proving memory wasn't decorative. My first working loop looked great — until I ran an ablation and found that disabling memory changed nothing: the agent reached the same diagnosis either way, because the alert and live state already gave away the answer. Memory was ornamental. Fixing this meant redesigning the fault itself so the root cause is not visible in live state — where only a past incident supplies it. I made "memory changes the outcome" a permanent, gating test: if the memory and ablated runs don't diverge, the build fails.

Distinguishing plumbing from reasoning. A demo can pass its pipes and still prove nothing about whether the agent actually thinks with its memory. Several bugs were silent degradations that looked like bad reasoning — an empty MCP path POSTing to a login page, a statement_timeout shorter than the cluster's lease-recovery window so every retry timed out mid-failover. Measuring, not assuming, was the theme throughout.

Being honest about resilience. CockroachDB survives a node loss, but a dead leaseholder needs a lease transfer — a measured few-second blip, not "zero interruption." I chose to demonstrate and state that honestly ("served straight through the failure, driver failed over") rather than overclaim something a judge could falsify live.

Amazon Bedrock access. Our AWS account hit an account-level authorization gate on Bedrock model invocation (ValidationException: Operation not allowed) — confirmed across every model, region, and credential form, lifted only by a support case. Rather than block on it, I leaned on the pluggable brain: the AWS requirement is met by S3, and the demo runs on Gemini, with the Bedrock backend ready for when access clears.

Accomplishments that we're proud of

  • Memory that is provably load-bearing — a gating ablation shows the agent fixing an incident because of memory and failing without it, on identical inputs. Most "agent memory" demos can't prove this.
  • A genuinely hard diagnostic scenario where every live-state heuristic is a trap, grounded in a real CockroachDB behavior (the lock table not exposing the holder).
  • The right architecture, not the convenient one — MCP for reads, a strongly-consistent transaction for writes, embeddings living beside operational data, and remediations kept safe by a strict whitelist.
  • Honest engineering under pressure — measuring failover blips, distinguishing plumbing from reasoning, and stating tradeoffs plainly rather than papering over them.

What we learned

  • "Uses a database" and "the database is load-bearing" are completely different bars. The ablation was the single most valuable thing I built, because it forced the difference into the open.
  • CockroachDB's integrated vector + strong consistency is a real advantage — storing an incident's embedding and its structured resolution in one transaction, in one system, is something you can't do cleanly with a bolt-on vector store.
  • Verify capabilities against docs and against the running system, never from memory — model IDs, index syntax, MCP behavior, and provider wire formats all had surprises that only surfaced by testing.
  • Trust in a tool comes from restraint and honesty. For an SRE audience, a clean console and truthful claims beat spectacle every time.

What's next for NineLives

  • Real incident detection — ingest alerts via webhook from the monitoring stack teams already run (PagerDuty, Datadog, Grafana), replacing the demo's injected alerts.
  • Human-in-the-loop approval for higher-risk remediations, extending the existing whitelist model.
  • Recurrence and alert de-duplication using the same signature similarity that already powers recall.
  • Chat integration (Slack / Teams) to triage and resolve from where teams work.
  • Beyond the database layer — the loop is domain-agnostic; the demo focuses on database incidents to showcase CockroachDB, but the same store → recall → act → remember pattern generalizes to any incident type.

Built With

Share this project:

Updates

Submission history