Swarm brain

• 

  logo

• 

  swarm-brain

Inspiration

Running more than one coding agent on the same repository fails in three specific, expensive ways. We have watched all three.

They collide. Two agents pick the same issue, edit the same file, and one silently overwrites the other. Every coordination scheme we tried-a shared markdown file, a task queue, a lock directory-either left a race window or required a human to become the lock.

They forget across the process boundary. An agent spends forty minutes finding a root cause, then its harness crashes or its context window rolls over. The finding dies with the process, and the next agent starts the same investigation from zero. Memory that lives only in a model vendor's session is memory the swarm does not own.

They poison each other. Once agents can write to shared memory, a confident, unsupported claim from one agent can become the premise of every other agent's reasoning. "Append everything an LLM says to a vector store" is not a memory system. It is a shared hallucination buffer.

All three are database problems wearing an AI costume. Safe claiming needs a serializable transaction and a uniqueness invariant. Crash recovery needs a lease, a fencing token, and a durable checkpoint. Safe shared memory needs evidence, bitemporal history, and a write policy. CockroachDB provides those properties in one distributed system, with SERIALIZABLE as the default-so we put the swarm's whole durable brain in it.

What it does

Swarm Brain is a vendor-neutral coordination and temporal-memory kernel for heterogeneous agent fleets. Claude Code, Codex CLI, Gemini CLI, and a local Qwen through OpenCode can all connect through the same local stdio MCP bridge. The bridge exposes nine model-visible tools through one HTTP API; CockroachDB holds all durable coordination, memory, event, and work-queue state. There is no separate lock service, message broker, or vector database.

One command runs the complete scenario and writes an auditable JSON artifact of named checks:

uv run --extra serve swarmbrain-demo                # in-memory, controllable clock
uv run --extra serve --extra crdb swarmbrain-demo   # durable CockroachDB

The reproducible four-vendor scenario uses scripted protocol drivers carrying Claude Code, Codex CLI, Gemini CLI, and OpenCode/Qwen roster identities over the canonical API. It proves the coordination protocol, not the reasoning quality of four live models. Separately, a real Claude Code process completed the full claim → recall → publish → checkpoint → complete loop through the production MCP bridge. Both are preserved as artifacts: the four-vendor CockroachDB run and the real-harness run.

The main demo proves eight behaviors in sequence:

1. Four vendor-labelled agents join one run.
2. All four race for two ready Wave-A investigations, and exactly two win. The other two receive no_task_available; the dependency-blocked Wave-B tasks remain unclaimable. One SERIALIZABLE transaction and a partial unique index enforce one active lease per task.
3. The winners publish two evidence-backed findings. Each registers a source, attaches a content-hashed excerpt, and publishes confirmed memory. Both waiting agents recall those findings with their evidence attached.
4. A correction succeeds and a poisoning attempt fails. Current recall returns only the evidence-backed correction; historical recall retains both versions and their lineage. An unsupported tentative supersession is a policy noop, leaving the confirmed memory current.
5. Wave A completes once and unlocks Wave B. Repeating a completion returns the stored result-same completion ID, no duplicate event or outbox row-and the dependent tasks become claimable only after their evidence lands.
6. A worker dies and a different vendor-labelled worker resumes it. A hidden verifier first proves that the supplied memories are causally necessary. The successor receives the expired worker's checkpoint and cited memories; the stale worker's late write is fenced with 409 lease_lost.
7. The two-wave DAG finishes in dependency order. The two agents that waited in Wave A become the Wave-B owners, and all 4/4 tasks complete.
8. The database reconstructs the whole story. The run exposes 26 durable events plus separate counts for a crash handoff, an idempotent replay, activation, citation, and cross-agent memory use.

We also rehearsed node loss against the same eight beats. A script started a fresh three-node CockroachDB 26.2.1 cluster and SIGKILLed a non-gateway node after beat 2 during live writes. All six remaining beats finished on the surviving quorum; zero ranges were unavailable; both survivors returned identical counts; and 4/4 tasks, three memories, and 26 events remained readable (artifact). This proves non-gateway node-loss resilience on one host. It does not claim gateway failover, disk-loss recovery, or multi-region behavior.

Recall uses exact-identifier, full-text, trigram, and dense-ANN lanes inside one MVCC snapshot. Weighted reciprocal-rank fusion combines them, the strongest results seed a bounded one- or two-hop memory graph, and a final hydration pass rechecks tenant, repository, visibility, lifecycle, bitemporal validity, and source trust before returning a hit. A targeted no-match regression returns hits=[] instead of a zero-score guess.

Who it's for

Swarm Brain is for platform teams deploying agent fleets and being asked three questions by finance, security, and procurement.

"What is this costing us?" In the measured coordinated run, agents used 25 steps. A disclosed deterministic baseline-same tasks and step costs, but no cross-vendor coordination, shared recall, or durable checkpoints-required 103 steps. That is 4.12× the work, or 78 additional steps. The baseline is a model, not a second live API run; the artifact records both arms and their assumptions.

"Can we audit it?" Bitemporal memory answers: which agent knew what, when, and on what evidence? Corrections append a new version and close the previous system-time interval; they do not rewrite history. Rejected poisoning attempts are recorded rather than hidden. Compliance can use a read-only, audit-logged CockroachDB Cloud Managed MCP connection that is separate from the enforcing write path.

"Are we locked in?" The durable brain lives in the organization's database, not in one model vendor's session. A worker can disappear and a worker carrying a different provider identity can resume from the same fenced checkpoint. The model or harness can change without discarding the fleet's operational memory.

Retrieval has measured evidence too. On all 500 questions in the cleaned LongMemEval-S release, the hybrid retriever reached Recall@10 0.970 and MRR@10 0.908. These are retrieval-only results over the benchmark's candidate sessions, not end-to-end question-answering accuracy. On a 40-query swarm-native corpus running on CockroachDB, C-SPANN returned the same top-ten set as an exhaustive exact-vector oracle in every case, while reducing p50 lookup latency from 45.2 ms to 15.4 ms. The raw runs and reports are in benchmarks/retrieval/.

Coding agents are the demo vertical because they are concrete and self-hosting; the claims, leases, checkpoints, evidence, and memory contracts are not coding-specific. The same kernel can coordinate support, operations, or claims processing fleets. The hosted hackathon deployment is single-region; REGIONAL BY ROW remains roadmap.

How we built it

CockroachDB holds the entire durable state machine:

• Coordination. tasks, dependencies, leases, checkpoints, and completions live together. Claiming expires elapsed leases, selects a dependency-ready task, and writes the lease, task transition, event, and outbox record in one short SERIALIZABLE transaction.
• Fencing and idempotency. Every mutation carries the expected task and lease version. A revived worker cannot overwrite its successor. An idempotency ledger stores a normalized request hash and response, so a retry replays while key reuse with a different command returns 409 idempotency_conflict.
• Bitemporal memory and evidence. Memories carry valid time, recorded time, lifecycle state, trust, review status, and visibility. Supersession is append-only. Source excerpts are verified character-for-character against preserved source material; rejecting a source fails closed for ordinary recall and retracts derived state in the same audited transaction.
• Hybrid retrieval. Exact terms, deterministic search documents, stored TSVECTOR, trigram text, and VECTOR(1024) projections are maintained with canonical writes. Projections accelerate candidate generation but never become an authorization boundary.
• Signed vector projections. Every vector records its canonical version, content digest, scope, renderer, model, dimensions, metric, normalization, and truncation policy. A C-SPANN index uses the complete seven-column equality prefix before embedding vector_cosine_ops.
• Durable background work. A separate swarmbrain-worker leases embedding and extraction work from transactional outbox tables. A worker that loses its lease cannot publish a vector, projection, effect, or completion.

The application layer is intentionally small: Python 3.11+, strict Pydantic v2 models, narrow async ports, FastAPI transport mapping, and an in-memory kernel implementing the same interfaces for deterministic tests. Backend selection is fail-closed: SWARMBRAIN_BACKEND must explicitly be memory or cockroach, and API startup verifies-but never installs-the schema. Embeddings use a port with a deterministic test provider and an Amazon Titan Text Embeddings V2 adapter at the schema's fixed 1024 dimensions. Database connections use TLS verify-full; public HTTPS for the demo API is pending a domain.

Challenges we ran into

Filtered ANN became a systems problem. CockroachDB vector indexes accelerate only a complete equality prefix, while our authorization model allows several repository, run, and task scopes. We issue one fully bound ANN branch per allowed scope, validate version, lifecycle, bitemporal, digest, and trust predicates against canonical rows in the same snapshot, and widen the ANN window geometrically when filtering under-fills the lane budget.

One failed lane can poison the transaction. Four retrieval lanes share one MVCC snapshot, but a failed statement can leave the transaction in 25P02. We serialize lane work on the connection and isolate every lane behind a savepoint. A live regression injects 42703 into one lane and verifies that the other lanes and final hydration still work.

SERIALIZABLE requires disciplined retry semantics. A 40001 reruns the whole transaction body, so mutation closures contain database work only and produce stable external identities. A 40003 means the commit result is truly unknown; we resolve it through the idempotency ledger instead of blindly repeating. Concurrent uniqueness conflicts (23505) enter the same resolution path.

The obvious graph and migration designs were not safe enough. Recursive CTEs were harder to optimize and bound than a fixed-depth traversal with explicit hop, fan-out, and edge-examination budgets. Large vector backfills must land before the vector index is created, and the atomic O(N) rebuild can retry, so schema installation remains an operator-planned maintenance action behind a writer barrier rather than an online migration.

Accomplishments that we're proud of

• A real claim race: four contenders, two ready tasks, exactly two leases, and no dependency bypass.
• A cross-provider-identity crash handoff: durable checkpoint recovery plus a fenced late write from the expired worker.
• A memory system that can say no: unsupported tentative advice cannot supersede confirmed, evidence-backed state, but the rejected attempt remains auditable.
• Evidence-backed retrieval: bitemporal correction, historical lineage, exact-vector parity checks, and explicit no-match behavior.
• A large live test surface: 1182 passed, 7 skipped against a disposable CockroachDB 26.2.1 node, plus 1143 passed, 46 skipped offline with no database or AWS. Live coverage includes EXPLAIN gates for every retrieval lane, transaction-poisoning isolation, projection rebuilds, stale-row sweeps, and temporal-decoy tests.

What we learned

• Serializable isolation is an agent-memory feature. Claiming, fencing, idempotent mutation, and atomic supersession become short transactions that either fully happen or do not.
• Write policy matters more than ranking. A better retriever over a poisoned corpus returns better-ranked poison. Evidence, trust states, and "tentative cannot supersede confirmed" improved safety more than another retrieval lane.
• Bitemporality makes correction safe. An agent can be wrong, be corrected, and still leave the exact history a human auditor needs.
• EXPLAIN and retrieval quality are different gates. One proves index selection; the other requires judged corpora and exact oracles. We test both.

What's next for Swarm Brain

• Broaden retrieval benchmarks with lane ablations, latency distributions, and filtered-ANN recall by scope-selectivity bucket.
• Build a resumable, rate-limited projection backfill with progress telemetry.
• Add bounded historical semantic recall without weakening current-state filtering.
• Persist audited retrieval traces and lane freshness/underfill metrics.
• Move tenant data to REGIONAL BY ROW and add more harness configurations.

Agent fleets will change models, vendors, and harnesses. Their coordination history and institutional memory should survive those changes. Swarm Brain makes that state durable, evidence-backed, auditable, and owned by the organization running the fleet

Built With

• amazon-bedrock
• amazon-titan-embeddings
• aws-ecs-fargate
• cockroachdb
• fastapi
• full-text-search
• mcp
• pg-trgm
• psycopg3
• pydantic
• python
• sql
• uv
• vector-search