Roshambo

„Let them play roshambo by the rules: many agents throw, only one can win. Serializable leases and failure memory on CockroachDB — proven across two machines and five vendors."

Comment

Inspiration

We run a local multi-agent shop — Claude, Gemini, Kimi and more — working the same repositories on the same machines. Two failures kept repeating.

The first is a locking story. Our own earlier, public tool lock-master coordinates agents through lock files — race-safe on one machine, broken across a synced folder. On 2026-07-23 a sync service forked our shared log, and each half kept a line the canonical file had lost. One machine, one referee. But our agents play on many.

The second failure is the interesting one. Existing agent memory stores what worked. Nobody stores what failed — exactly the knowledge that separates an experienced engineer from a freshly spawned agent.

So we built a referee that also keeps score: a serializable transaction calls every throw, and a vector index remembers every defeat.

What it does

Roshambo is a coordination and failure-memory fabric for heterogeneous agent swarms, with CockroachDB as the system of record. Every unit of work is a throw of rock-paper-scissors: any number of agents may reach for the same task, exactly one wins — and every loser is told who beat them, and what the winner is doing.

Three verbs around every unit of work:

  • claim(resource, intent) — a serializable lease. Denial is a feature, not an error: the caller learns who holds it and what they are doing.
  • remember(topic, approach, outcome, evidence) — records how an attempt ended. Failures are first-class records.
  • recall(query) — vector search over past attempts, including failures, on CockroachDB's distributed vector index. Honestly stated: proven with the offline lexical embedder; the semantic-recall test against Bedrock Titan is blocked by this account's quota and has never completed (docs/EVIDENCE-bedrock.md).

Why not AgentCore Memory or Claude Code Agent Teams? Both do parts of this well. One is a memory service for an agent; the other shares state inside one vendor's sessions. Neither lets a Claude agent, a Gemini agent and a Lambda worker — strangers sharing nothing but a database — avoid claiming the same work twice.

Why shared state, and not a protocol

Roshambo has no message format two agents must both implement. A protocol needs both sides to speak it; shared state only needs each side to read it. That is why agents from five vendors coordinated in our run with zero adapter code per vendor — and why a 200-line script bot sits at the same table as an LLM agent.

Who locks the lock file? In the file world that regress ends at O_EXCL, atomic only locally. With a database it disappears: INSERT … ON CONFLICT (swarm_id, resource) against the primary key is the mutual exclusion, supplied by the serializable transaction underneath — one layer below the tool, and it holds across machines.

How we built it

Two properties have to live in one database or they drift apart: leases (serializable transactions) and recall (a vector index). The competition text names this directly — "no consistency gaps between your vector data and your operational database." Architecture diagram: README.md, "Architecture".

Validated the hard way: a closed two-machine field run (poc-starmap-2026-07-30) — 27 participants from Anthropic, Google, Moonshot, Zhipu and MiniMax plus eight deterministic script bots, building an invented star chart in one shared repository. Twice the setup itself failed instructively: two DSNs differing only in the database name silently split the swarm, and the two machines pushed to two separate repositories before a shared remote merged them. Both failures are documented, not edited out (evidence repo: ellmos-ai/roshambo-starmap).

CockroachDB tools used (all four in the repo; three run and evidenced):

  • Distributed Vector Indexing — trails/facts carry VECTOR(1024) columns with a swarm-prefixed VECTOR INDEX; recall() runs cosine search (<=>) per swarm. Run against a local node and CockroachDB Cloud (docs/EVIDENCE-core.md, docs/EVIDENCE-cloud.md).
  • Cloud Managed MCP Server — read-only inspection, deliberately separate from our write-path MCP server. A real session on 2026-07-31: list_databases, get_table_schema, select_query against the field run's audit_log — verb counts matching collect_evidence.py exactly (docs/EVIDENCE-mcp.md).
  • ccloud CLI — provisioning wrapper written, check subcommand run; provisioning itself never run against a real login. Built, not demonstrated — stated plainly.
  • Agent Skills Repo — the official cockroachdb-sql skill followed step by step against our live schema: it cleared claims and flagged a genuine sequential-key hotspot on audit_log's primary key (docs/EVIDENCE-skills.md).

AWS services used:

  • AWS Lambda — the demo web app runs as a public Function URL in eu-central-1, same region as the cluster; least-privilege IAM, budget guard. Live below.
  • Amazon Bedrock — Titan Text Embeddings V2 is the production embedder path. One verified live InvokeModel call proves the integration; semantic recall is quota-blocked and disclosed as unproven (docs/EVIDENCE-bedrock.md).
  • Amazon S3 — large payloads referenced by s3:// URI (trails.artifact_uri), never stored inline. Code path implemented, not yet exercised against a real bucket.

Live demo

https://xo7te46ion5mhwi6mhua6va7im0cotkk.lambda-url.eu-central-1.on.aws/ — public, read-mostly, no credentials by design. /api/health answers {"mode":"live"} against the real cluster. Cold start ≈2.3 s billed, warm 2–13 ms (CloudWatch, demo/README.md).

The live map → — the replay viewer's projection pointed at the present: agents grouped by host, active leases as chips on their holders, the audit log as a live feed. One read-only snapshot endpoint, polled. This is the "satellite GPS for agents" from What's next — built and deployed during the submission window.

The strongest proof — a judge can run it without an LLM key

Exactly one winner is a deterministic property of one SQL statement, checkable in thirty seconds against any free CockroachDB Cloud tier:

python demo/multivendor/bot_agent.py --dry-run --bots 3

Measured 2026-07-31: 24 attempts by 3 bots, 10 granted, 14 denied, 0 errors — "two simultaneous holders of the same task, at any point: no".

At scale: 1,226 genuine collisions in 4,011 audit rows, 147 across the machine boundary, 1,245 of 1,245 denials naming the exact current holder — never two live leases. The detector's only three anomalies were reconstructed row by row: audit-write lag, not double grants (docs/EVIDENCE-defects.md).

Challenges we ran into

  • A wrong vector index operator class fails silently. CREATE VECTOR INDEX defaults to vector_l2_ops; a cosine <=> query full-scans without error — invisible until EXPLAIN. Fixed by pinning vector_cosine_ops; reported to Cockroach Labs.
  • Two DSNs differing only in the database name split the swarm in two — same cluster, zero shared state, invisible from outside. Later the same mistake one level up: two machines, two separate star-chart repositories.
  • Measuring "two agents hold this at once" is easy to get wrong — we got it wrong twice. Network latency inside a detector window; a checker that ignored releases and counted 134 phantom overlaps. Both wrong detectors documented, not deleted.
  • A public Lambda Function URL needs two IAM permissions since October 2025, not one — lambda:InvokeFunctionUrl alone returns a silent 403.
  • A model listed for a region is not necessarily callable there — Titan V2 lists in eu-central-1; the invocation quota reads 0.
  • TLS to cloud CockroachDB needs an explicit root certificate even for a public CA — libpq does not use the OS trust store.
  • A skills catalogue can contain empty shelves — three official skill directories held only .gitkeep; those tasks were withdrawn, not improvised.

Accomplishments that we're proud of

  • 27 agents, five vendors, two machines — exactly one winner, every time: 1,226 collisions, 147 cross-host, 1,245/1,245 holder-named denials, never two live leases.
  • A failure recorded by one agent is found by another through a differently worded query (rank 3 unfiltered, rank 1 restricted to failures, distance 0.6113 — plainly stated, not dressed up as semantic search).
  • A working demo, public and live right now — not a screenshot.
  • 189 tests pass, 50 skip cleanly without cloud credentials (pytest -q, with the demo extras from demo/requirements.txt installed).
  • The live map (/live) shipped after the first submission edit: one read-only snapshot endpoint, adversarially reviewed before deployment — the review caught a real leak (visitor search text would have been broadcast to every map watcher) and it was fixed server-side before the endpoint ever went live.
  • One genuine, credentialed Bedrock call returned a real 1024-dim vector — integration proven, semantic recall honestly unproven.
  • Both repositories public and cross-linked — tool (Apache-2.0) and evidence (MIT), including our own mistakes in the reports. Nothing staged, nothing smoothed.

What we learned

  • "No overclaiming" is a checked habit, not an intention. Every load-bearing sentence cites its evidence file; "can't find where this was measured" means soften the sentence.
  • Shared state beats a protocol across vendors — no vendor-specific branch anywhere in the coordination path, and a script bot coordinates exactly like an LLM agent.
  • A proof that shows its own mistakes is worth more than one that doesn't. Ours are in the repos on purpose.
  • Regional AWS grants are not one setting — listed ≠ callable.

What's next for Roshambo

CockroachDB as satellite GPS for agents — built: the live map is deployed as part of the demo app, and it was exactly the projection change over the claims table this paragraph predicted, not a schema change. Still open on this line: presence beyond heartbeat recency, and a fleet view across swarms. Agents spawned from AWS — the packaged roshambo-worker Lambda, deployed for real. The team-message table — directed agent-to-agent chat as claimable shared state, crossing machine boundaries because it lives where the leases live. Plus per-agent authorization on the write path, the walk-in 3D planetarium for the evidence repo's sky, and closing the one open verification gap: a Bedrock quota increase, so semantic recall can finally be measured instead of only attempted.

Roshambo — Links

What Link
Tool repo (public, Apache-2.0) https://github.com/ellmos-ai/roshambo
Evidence repo (public, MIT) https://github.com/ellmos-ai/roshambo-starmap
Live demo (AWS Lambda) https://xo7te46ion5mhwi6mhua6va7im0cotkk.lambda-url.eu-central-1.on.aws/
Live map (agent GPS) https://xo7te46ion5mhwi6mhua6va7im0cotkk.lambda-url.eu-central-1.on.aws/live
Run replay simulation (online) https://ellmos-ai.github.io/roshambo-starmap/replay/
Planetarium (online) https://ellmos-ai.github.io/roshambo-starmap/planetarium.html
lock-master (predecessor) https://github.com/dev-bricks/lock-master

Built With

Share this project:

Updates