Ninelives

An AI research agent whose entire mind lives in CockroachDB. Kill it mid-word — a standby in another AWS region resumes the same sentence in seconds, having forgotten nothing.

Comment

Inspiration

"Everything fails, all the time." Today's AI agents keep their minds in process memory — kill the process and they restart with amnesia, back at step one. We wanted to invert that: make the database the mind, so the agent itself becomes disposable. CockroachDB is built to survive; an agent whose every thought lives there should survive too.

What it does

Ninelives is a long-running AI research agent whose entire cognitive state — task progress, partially-written sentences mid-stream, and episodic memory — lives transactionally in CockroachDB. There is no state anywhere else.

The live arena shows two workers in different AWS regions (us-east-1 and eu-central-1) and a KILL AGENT button anyone can press. Kill the active worker mid-word and the standby claims the lease and resumes the same sentence in seconds — measured at 0.74–3.39s across six consecutive live rounds. A brain monitor streams every memory write in real time (CockroachDB changefeed → SSE), and a recall panel answers questions from episodic memory via vector search, with SQL provenance rows for every answer: the model can be wrong, the rows cannot.

As a contrast, the same agent run with memory disabled dies exactly the way you'd expect: total amnesia.

How we built it

CockroachDB is load-bearing, not a bolt-on vector store:

  • Distributed Vector Indexing — episodic memory (memory_events) with embeddings, powering sub-second recall with provenance.
  • Managed MCP Server (read-only) — our live operations window into the agent's mind during development; it even caught a runaway process for us.
  • ccloud CLI — cluster provisioning and ops.
  • Core changefeeds stream every memory write to the brain monitor UI.
  • Every agent step is transactional: lease claim/renew under SERIALIZABLE isolation (with the documented retry-on-40001 client contract), streamed LLM output persisted chunk-by-chunk, embeddings inserted, step results committed — so a kill -9 at any instant loses nothing.

AWS: two ECS Fargate services run the workers in us-east-1 and eu-central-1, a third runs the arena behind an Application Load Balancer, with an ACM certificate on the public demo URL. The LLM is Claude via the Anthropic API; embeddings are local sentence-transformers — both behind a one-line provider switch.

Challenges we ran into

  • Real concurrency bugs only live kills could find: concurrent lease access under SERIALIZABLE crashed the primary with a serialization failure (fixed with CockroachDB's documented retry contract), and a lease TTL could expire during the model's thinking latency, letting a healthy primary get evicted (fixed with a dedicated heartbeat thread that dies honestly with its process).
  • An intermittent 25s failover stall that looked like a database problem — and wasn't. We instrumented a live stall with crdb_internal.cluster_locks and SHOW SESSIONS: zero database activity the whole time. The real cause was ECS Fargate task replacement (~170s measured) racing a too-short kill cooldown. The database was never the bottleneck; evidence-first debugging saved us from "fixing" the wrong layer.
  • Container-only bugs: a PID-1 SIGKILL no-op, a silently re-installed CUDA build, a missing CA cert for verify-full TLS — all found by running the real container against the real cluster.

Accomplishments that we're proud of

  • A juror can kill our agent themselves, right now, at the live URL — and watch it refuse to die.
  • Six consecutive cross-region kill-and-resume rounds, all under 3.4 seconds, nothing simulated: what the demo shows is what the database did.
  • The provenance design: every recall answer arrives with the SQL rows that ground it.

What we learned

SERIALIZABLE isolation plus the client retry contract is what makes "a mind in a database" actually safe with two claimants racing for it. And in distributed systems, measure before you blame a layer — our worst bug lived in orchestration, not storage.

What's next for Ninelives

Multiple agents sharing one durable memory store; temporal reasoning over episodic memory; and Amazon Bedrock as an alternative model provider behind the existing config switch.

Honest notes

Built with AI coding assistance (Claude Code). The research corpus is curated and bundled, not live-fetched. LLM calls go to the Anthropic API; embeddings run locally. Resilience, persistence, and failover are never simulated.

Built With

Share this project:

Updates