HiveMind

A swarm of AI agents that coordinate through CockroachDB. They claim work with transactions, share one memory, and recover when an agent dies mid-task. No central orchestrator holds it together.

Comment

HiveMind — the database is the coordinator

Inspiration

Every multi-agent system I looked at had the same shape: one orchestrator handing work to N agents. Draw it and every arrow points out of a box in the middle. That box is a bottleneck, a single point of failure, and the part nobody demos — because when it dies, the demo is over.

Distributed systems solved this decades ago, and the answer was never "a better orchestrator." It was choreography instead of orchestration. CockroachDB is serializable by default and has no central coordinator in its own architecture, so the question that started this project was: if the database already gives me serializable transactions across a distributed cluster, what is the orchestrator actually for?

The answer turned out to be nothing.

What it does

An alert hits a Lambda, which explodes one incident into $N$ independent task rows in a single transaction and returns 202. Identical, stateless agents — five, or fifty — claim those tasks straight out of the table, investigate in parallel with Claude on Bedrock, retrieve semantically similar past incidents from a vector-indexed memory table, and synthesise a diagnosis that gets written back into memory so the swarm is better at the next one.

Work is pulled, never pushed. Nothing assigns anything. Nothing detects failure. Then you kill -9 an agent mid-task on camera, and the work finishes anyway.

The diagnosis is the setting. The coordination is the system.

How we built it

Three thin layers, all of them tables.

Coordination — tasks. The whole protocol is four SQL operations. The claim is the load-bearing one:

UPDATE tasks
   SET status = 'claimed', owner = $1,
       lease_expires = now() + INTERVAL '10s',
       attempts = attempts + 1
 WHERE id = (
         SELECT id FROM tasks
          WHERE status = 'pending'
             -- reclaim work abandoned by an agent that died holding it
             OR (status = 'claimed' AND lease_expires < now())
          ORDER BY priority, created_at
          LIMIT 1 FOR UPDATE SKIP LOCKED
       )
RETURNING id, kind, payload, attempts;

Two invariants make it safe: claims are exclusive, and every later mutation is guarded on owner = me AND status = 'claimed', so an agent that lost its lease affects zero rows and must discard its work. Self-healing falls out of that OR clause — "abandoned by a dead agent" and "never started" are the same condition to the query. Recovery is the absence of a code path, not the presence of one.

Memory — incidents. VECTOR(1024) with the index prefixed on service, so EXPLAIN shows a vector search with prefix spans rather than a full scan. A test asserts that plan.

Audit — agent_actions, written in the same transaction as the operation it describes, so the log can never record a claim that was rolled back.

On top: Bedrock (Claude + Titan v2 embeddings), Lambda for the alert entrypoint, CockroachDB's managed MCP server for the agents' read path, ccloud for provisioning and RBAC. Reads go through MCP because it's server-enforced read-only and audit-logged; writes go over direct SQL because a claim is a transaction and can't survive being decomposed into tool calls.

Challenges we ran into

  • The chaos harness kept killing itself. It SIGKILLs a random node; its own admin connection lands on a random node. With three nodes that's the same node about $1/3$ of the time — the observer died while the swarm it was measuring carried on working perfectly.
  • A test that proved nothing. libpq walks multi-host DSNs in order, so without load_balance_hosts=random all 8 agents connected to node 1 — the node-failure test was killing a node no agent was on, and passing.
  • claim_one() returning None doesn't mean the queue is empty. Under SKIP LOCKED it also means "everything left is locked right now." Invisible on one node; on three, 50 agents claimed 2 of 20 tasks and two tests called it drained.
  • The database quietly stopped accepting concurrency. An unlicensed cluster is throttled to five concurrent transactions after a 7-day grace period, which makes a 50-agent test impossible. Nothing in the error says "licence."
  • The demo poisoned its own memory. Completed incidents write their diagnosis back — so re-firing the same alert retrieved the swarm's own prior output at distance 0.000, and take two always looked like a cache hit.

Accomplishments that we're proud of

Under chaos, the numbers held. 600 tasks, 50 agents, with 43 agents and a database node SIGKILLed mid-run:

completed  600/600     COMPLETED TWICE  0     reclaimed from a dead agent  18
claim p50  20.8 ms     p95 101.9 ms           p99 234.6 ms

A killed agent's task was reclaimed 9.7s into a 10s lease and finished by someone else. 98 tests, all against a real 3-node cluster — never mocks, which would only agree with our own assumptions. And an agent can't read the audit log of its own actions (HTTP 403), with a test that fails if anyone over-privileges the service account later.

What we learned

The thing we were most confident about was wrong. We believed FOR UPDATE SKIP LOCKED prevented double-claims. Mutation testing removed it entirely and still measured zero double-claims across 50 agents × 20 tasks × 5 runs. Correctness comes from serializability; SKIP LOCKED buys throughput — without it $n$ claimers convoy on one hot row into $\Theta(n)$ behaviour, measured at 3.5×. Two different properties, and we'd been conflating them.

Client clocks are nowhere in the safety path. lease_expires < now() is evaluated by the database, so all leases share one clock. Liveness wants $T_{\text{renew}} + \delta_{\text{net}} < T_{\text{lease}}$, with recovery bounded by $t_{\text{recover}} \le T_{\text{lease}} + T_{\text{poll}}$ — but violating it costs duplicate execution, never duplicate completion, because the owner guard rejects the loser's write. Bad timing costs money, not correctness.

Normalized embeddings make the index choice a one-line proof. For unit vectors $|a-b|_2^2 = 2 - 2\cos\theta$, so L2 and cosine rank identically and the default vector_l2_ops index is correct — verified Titan v2 returns $|v|_2 = 1.0$ before relying on it.

Stubs must derive from real inputs. A random offline stub makes every retrieval meaningless while still looking like it works, so a handler passed the wrong data would be indistinguishable from one that works.

What's next for HiveMind

  • Task dependencies (depends_on), so an investigation can fan into a DAG rather than a flat set — deliberately cut to keep the primitive provable first.
  • Changefeeds instead of polling for the dashboard, so the board is pushed updates by the database rather than asking every 500ms.
  • Genuinely multi-region. The topology is real today but the three regions are simulated on one machine; the latency isn't.
  • Memory consolidation — as the corpus grows, near-duplicate incidents should merge rather than accumulate.

Built With

Share this project:

Updates

Submission history