HiveMind

A distributed memory and control plane for AI agent fleets, built on CockroachDB - proven by a fraud-investigation swarm where losing state means losing money and every decision must be auditable.

Comment 1

Inspiration

The CockroachDB × AWS hackathon asked for agentic memory that actually holds up in production. Most agent memory demos are a vector store bolted onto a chatbot - forgiving if it's slow, forgiving if it loses a write, forgiving if two writers race. We wanted to build the opposite: an agent fleet where losing state means losing money, and every decision has to survive an audit. Fraud investigation over PaySim's transaction stream was the harshest workload we could think of to prove agentic memory on CockroachDB isn't just a toy - it's the same guarantees a bank would demand.

What it does

HiveMind scores a stream of payment transactions; the ~2% a classifier can't clear go to a fleet of stateless Lambda agents. Each agent claims a case with SELECT ... FOR UPDATE SKIP LOCKED, recalls similar past cases from a shared episodic memory (CockroachDB's vector index), reasons with Claude on Bedrock, and returns fraud / legit / escalate - writing an append-only audit trail for every step. Crashed agents resume mid-case from checkpoint. A human reviewer only sees the cases the fleet itself flags as uncertain, and every human decision - single or bulk - gets embedded back into the same memory a pinned, undecayable lesson. Operators drive the whole system from a live control-plane dashboard: feed data, watch verdicts, kill an agent mid-investigation and watch it recover, add a CockroachDB region and watch survival goals update live.

How we built it

  • CockroachDB is the only datastore: the Managed MCP Server gives the agent a read-only SQL exploration surface (least-privilege at the protocol layer), a partial VECTOR INDEX (WHERE archived = false) powers episodic recall and shrinks as salience-based forgetting archives stale memories, and multi-region SQL with SKIP LOCKED coordinates 20+ agents with no broker and RPO = 0 on a region loss.
  • AWS Lambda (7 Go container-image functions) does all compute, scaling to zero at idle; EventBridge schedules the dispatcher, a heartbeat reaper, and salience decay; S3 + CloudFront hosts the static Next.js control-plane dashboard.
  • Amazon Bedrock: Claude Haiku reasons over each case, Titan Embeddings v2 vectorizes it for recall - deliberately a small, cheap model, made trustworthy by a deterministic categorical tool (balanceSignal) that traces the money flow, so the model only has to say what it can prove.
  • Terraform provisions everything from zero (./scripts/init.sh); GitHub Actions runs a full CI/CD chain - hermetic tests, a real integration suite against a disposable CockroachDB container, security scans, staged deploy, smoke test, then a canary that checks its whole 5-minute observation window before auto-promoting or rolling back.

Challenges we ran into

  • A real CockroachDB scheduler bug, not ours: under our exact claim pattern, SELECT ... FOR UPDATE SKIP LOCKED intermittently returned zero rows for a task that had just committed and genuinely existed - traced it to cockroachdb/cockroach#167582: SKIP LOCKED's scanner skips any key with an intent, including one from a transaction that already committed but hasn't been asynchronously resolved yet. Fixed with a short bounded retry inside ClaimNextTask rather than papering over it with a sleep in the test.
  • A silently-swallowed migration bug: human_lesson_stored - the audit action written the moment a human-taught memory gets pinned - was never added to audit_log's CHECK constraint. Every such write had been failing with a 23514 since the feature shipped, silently, because the call site discarded the error. Reproduced against a live CockroachDB container, fixed with a migration, and locked in with a regression test.
  • CloudWatch billing alarms in a multi-region Terraform setup: AWS silently requires a billing alarm's SNS topic to live in us-east-1 regardless of where everything else runs - the error message is identical whether the region argument or the SNS target is wrong, which cost real time chasing the wrong hypothesis before the actual fix (a dedicated us-east-1 SNS topic via a scoped null_resource).
  • Small-model trustworthiness: a cheap model asked to "is this fraud" free-form hallucinates confidently. Giving it one deterministic tool that answers a narrow, provable question (does the money trace to a clean destination?) turned out to matter more than model size - see docs/adr/0002.

Accomplishments that we're proud of

  • 100% recall and 100% precision on the labelled eval (34/34 fraud caught, 0/340 false positives), reproducible end-to-end with one command.
  • ~46% of escalated cases resolved without a human, at $0.00023 per investigation.
  • Zero double-claims across a 20-worker concurrency test, and thousands of real crash-and-resume cycles in this fleet's own audit history - verified live, not just in a demo script.
  • A dashboard that isn't a static mockup: every panel, including multi-region status and chaos-test controls, reads and writes the real control plane.

What we learned

That "agentic memory" is a systems problem before it's a model problem: the hard part wasn't getting an LLM to reason well, it was making sure 20 competing agents never double-claim a case, a crash never loses a checkpoint, and a human's correction is provably the same durable memory the fleet itself writes to - not a side channel. CockroachDB's vector index with a partial predicate turned "forgetting" into a query-planner concern instead of app-side filtering, which we didn't expect to matter as much as it did.

What's next for HiveMind

Wiring the MCP search_similar_cases tool onto the live investigation path (currently implemented and tested, but recall still goes through the vector index directly); extending the same memory-and-control-plane pattern beyond fraud to any domain where an agent fleet needs shared, durable, human-correctable memory - support triage, infrastructure incident response, content moderation - since nothing about the architecture is fraud-specific except the reasoning prompt and the PaySim feed.

Built With

Share this project:

Updates

posted an update —

Note for judges - dashboard link typo in the submission

The "live dashboard" URL in our submission description is an outdated address pasted by mistake when filling in the form (left over from an earlier deployment - it no longer resolves).

Correct dashboard, live since before the deadline: https://dt4qlrm6qkryj.cloudfront.net (the same URL shown at the end of our demo video)

The project itself - repository, demo video, code, and the running system - is untouched and unchanged since submission. Sorry for the mix-up, and thank you!

Log in or sign up for Devpost to join the conversation.

Submission history