FleetMem - agentic memory for robot fleets

Warehouse robots sharing one CockroachDB memory layer. Memory makes each agent competent and makes it physically impossible for two robots to claim the same dock.

Comment

Inspiration

In a traditional application, a lost write shows somebody a stale page. For an AI agent, memory is not a cache it is the input to the next action. So a lost or racy write does not degrade the output; it makes the agent do the irreversible thing twice. In a warehouse, that is not a duplicate email. That is a collision.

We went looking for a domain where that distinction bites hardest, and found one nobody was serving. AWS discontinued RoboMaker in September 2025. Robot fleets still coordinate through ROS 2 DDS - ephemeral pub/sub-rosbag files, and per-robot SQLite. Coordination frameworks prevent resource conflicts in process memory, so a fleet-manager crash evaporates every allocation.

A warehouse fleet is a partition-prone distributed system whose agents take irreversible physical actions and cannot accept a maintenance window. That is textbook CockroachDB, in a sector that has never heard the pitch.

What it does

Warehouse robots share one CockroachDB Cloud memory layer. Every dock is a physical resource that exactly one robot may hold. Each agent recalls what the whole fleet has learned before it acts, and the database decides whether its actuator is allowed to move.

  • Exactly one holder. A partial unique index - UNIQUE (fleet_id, resource_id) WHERE released_at IS NULL means at most one live claim per resource. The loser gets a deterministic 23505 carrying the current holder, so it re-routes instead of retrying into a dock that will never free. Drop the index, re-run the same race, and both robots get the dock. That is the collision, and it is why the constraint is a safety mechanism rather than a schema detail.
  • No dual-write. A lesson and its 1024-dim embedding commit in the same transaction. There is no window where the row exists and the vector does not. A separate vector store cannot close that gap, and its drift is silent distances still compute, answers still look plausible.
  • Isolation is structural. VECTOR INDEX (fleet_id, embedding vector_cosine_ops) puts fleet isolation in the index prefix, not a WHERE clause a refactor could drop.
  • Fencing tokens. A lease bounds the claim; it does not bound the machine. A robot paused by GC can wake after expiry still intending to move. Every grant issues a monotonic epoch, checked before every commanded motion the mechanism Chubby, ZooKeeper, etcd and Kubernetes use.
  • Memory has a lifecycle. Lessons carry event time and ingestion time, validity windows, and supersession. recall(as_of=...) answers "what did the fleet believe on Tuesday, and why did it act that way?" what incident reconstruction under ISO 3691-4 and ANSI/RIA R15.08 requires of an autonomous system.
  • It drives real ROS 2. ros2/fleetmem_bridge.py is a real rclpy node: subscribes nav_msgs/Odometry, publishes geometry_msgs/Twist, and calls the database between them. A denied claim publishes a zero Twist - the denied robot's wheels physically stop.

How we built it

CockroachDB Cloud (London, aws-eu-west-2) is the system of record: claims, fleet memory with a distributed vector index, durable agent checkpoints, and an audit trail.

AWS provides the intelligence and the hosting - Bedrock Titan Text Embeddings V2 for the 1024-dim vectors, Bedrock Amazon Nova Pro via the provider-agnostic Converse API for agent planning, S3 for bulk incident artifacts, and EC2 behind Caddy with automatic Let's Encrypt TLS.

A FastAPI service runs the simulation and streams state over WebSocket to a three.js console; a separate ROS 2 bridge runs the same memory layer against real robot topics.

Every claim is a serializable transaction. Serialization errors (40001) are retried with exponential backoff and jitter un-jittered retries re-collide in lockstep and reproduce the contention they are backing off from.

Challenges we ran into

Four of our worst bugs were silent - which is exactly what this project is about.

  1. The UI reported reasoning: bedrock while running a local fallback. The probe only checked that a boto3 client could be constructed.
  2. The fix then "verified" Bedrock by calling STS which succeeds with credentials that have no Bedrock permission at all. It lied more convincingly. Valid credentials are not model access. It now performs a real inference call.
  3. A totally failed race returned HTTP 200 with an empty list, because exceptions raised in worker threads never reach the caller.
  4. Adding connection pooling silently disabled our 40001 retry path in psycopg3 SerializationFailure subclasses OperationalError, so the new pooled connect() swallowed retryable conflicts and reported them as an outage.

Our own fencing implementation became the bottleneck it was meant to protect. The first load test leaked 202 serialization failures. The cause was SELECT MAX(epoch)+1, which forced every concurrent claimant to read the same rows the locking mechanism had become the dominant source of lock contention. Moving to a non-transactional sequence took errors to zero and throughput from 18 to 45 ops/sec.

Anthropic inference profiles were unavailable in our AWS account. Because we had built on the Converse API, switching providers was a config value rather than a rewrite.

Accomplishments that we're proud of

Everything below is measured, and each has a test that fails loudly if it regresses:

  • The race is proven both ways. With the index: one holder. Drop it: two holders, a collision. A gate nothing can fail is a bug wearing a safety costume.
  • Node kill mid-workload: 80 operations, 56 after the kill, 0 failures.
  • Load: 50 robots over 8 contended resources - 45 claim ops/sec, 0 errors, 0 double-holds, with the invariant asserted continuously during the run.
  • Connection pooling: 1065ms → 20ms median query.
  • Real ROS 2: a denied robot's integrated pose froze for eleven seconds and resumed the instant CockroachDB granted its claim.
  • Fencing proven through ROS 2: we stole a moving robot's claim; it logged FENCED: presented epoch 11, current is no live claim and its velocity dropped to zero on the next tick.

What we learned

A correctness mechanism added without a load test can degrade the very property it protects, and no functional test will reveal it. Our fencing tokens were correct and made the system markedly worse under contention.

Checking something adjacent to your claim is not checking your claim. All three false-reporting bugs shared that root cause constructing a client instead of calling it, calling STS instead of Bedrock, counting rows at the end instead of detecting overlap.

Production agent-memory research is converging on exactly this stack: vector search, structured queries for history and metadata, and ACID transactions when multiple agents share state. That is CockroachDB, described from the agent side rather than the database side.

What's next for FleetMem

  • Cross-container and multi-machine DDS discovery both ROS nodes currently share one container, which removed the highest-risk failure mode but leaves the distributed case unproven.
  • Widen the agent's output schema. It picks a target and a speed, so negotiation and preemption are out of reach for any model a design limit, not a model limit.
  • Multi-region topology. We exercise cross-region latency (London cluster, us-west-2 compute) but not a true multi-region cluster.
  • Outcome-weighted memory - record whether acting on a lesson actually helped, and rank recall by measured usefulness rather than cosine distance alone.

Built With

Share this project:

Updates