Backstop

Inspiration

Every mutual-aid group runs on one volunteer. They read every post, hold the whole thing in their head, and make the introductions. Their memory has a horizon of a few days.

So a request posted on a Tuesday scrolls off the channel by Thursday, unmet, and nobody sees it again, because by definition nobody is looking. That is not a hunch. On 5,671 real requests with known outcomes, 75% went unmet.

The volume is not what breaks. Follow-through is. And that is exactly the shape a machine holds well: it does not forget, and it does not stop looking. So the memory here is not a cache sitting in front of a model. The memory is the product.

What it does

Backstop watches a real community and remembers three things about every request: what was asked, what became of it, and how to find it again by meaning.

It reconstructs each request's fate. On a channel that tracks state it reads the platform's status; on one that tracks nothing it reads the settled thread. Validated against ground truth on 5,671 labelled requests: a need that drew no response is dropped at 95% precision, one with a named giver is resolved at 100%, and the ambiguous middle it marks uncertain and hands to a human rather than guessing.

Then it surfaces what fell through. On live Freegle data it found a television wanted in Edinburgh, still open after 262 days, and a request open for 914. Each one carries the post that proves it.

The memory layer

CockroachDB holds it. Every record carries its fate and a 1024-dimension Amazon Titan embedding in a VECTOR(1024) column behind a distributed vector index, so recall works by meaning rather than by keyword. Right now it holds 200 real posts from two Freegle communities, 30 of them dropped needs.

Same engine in both places: cockroachdb/cockroach single-node in the local compose stack, CockroachDB Cloud in production. The vector column and the index behave identically in development and in the deployed Lambda, which is the point of developing against the real thing rather than a stand-in.

How we built it

CockroachDB tools used: the Cloud Managed MCP Server, and distributed vector indexing.

AWS: Lambda hosts the agent as an arm64 container image behind a public Function URL. Bedrock runs the Titan v2 embeddings.

The agent itself is a Strands multi-agent coordinator over four tools: intake, the backstop sweep, the matcher, and a scribe that drafts follow-ups. Nothing posts unattended. Every action class starts at draft and earns autonomy one approval at a time, and a single rejection demotes it a rung.

Challenges we ran into

Vector search did not fix the bug we built it for. We had already published a failure: on real data our matcher offered a plain table for a pool table, pillows for a sofa, a cardboard box for a mobility-scooter shelter. Nine proposed, eight false. Embeddings looked like the obvious fix, since those pairs are close as strings and far apart in meaning.

The first honest test failed the same way. Asked for a pool table, the nearest offer was still a coffee table. Nearest-neighbour is obliged to return a neighbour. Embeddings were necessary and not sufficient.

What fixed it was the signal embeddings expose. Measured on the live corpus, splitting queries by whether the item is genuinely present:

query	nearest-neighbour distance
present (office chair, car seat, double duvet, thermos)	0.394 to 0.599
absent (pool table, heat pump, scooter shelter, grand piano)	0.734 to 0.817

No overlap, and a gap of 0.135. A threshold at 0.65 sits inside it, so the matcher can finally decline:

an office chair to sit on   ->  MATCH  Swivel Office Chair (0.519)
a car seat for a baby       ->  MATCH  Child's Car Seat    (0.599)
a pool table                ->  no match
a grand piano               ->  no match

The threshold is calibrated rather than chosen. That is the whole difference from the version that failed, which rested on prompt wording nobody had measured.

Being honest about what memory is for. It is tempting to make matching the headline. Real data says otherwise: on a community, a wrong introduction spends the one thing the group runs on, which is goodwill. So the spine is the sweep, which surfaces a dropped need and never asserts a pairing. Matching sits behind the threshold, a skeptic, and a human's yes.

Accomplishments that we're proud of

Recall that works by meaning on real data, with a measured threshold and a documented failure path rather than a demo that only shows the happy case. 49 tests. And a system whose default, when it cannot tell, is to say so and ask a person.

What we learned

A memory layer is not a database with the word memory in front of it. What made this one useful was storing the fate of each request alongside the request, and being able to retrieve by meaning months later. Neither is possible with a log.

And retrieval quality turned out to be a measurement problem, not a modelling problem. The embeddings were fine from the first call. What was missing was knowing where the boundary sat between a real match and nothing at all.

What's next for Backstop

Re-calibrate the threshold per community, since the constant transfers less well than the method does. Push more of the fate reconstruction through the MCP server so the agent can query its own memory conversationally. And put it in front of a real dispatcher, which is the only test that finally counts.

Built With

• agentic-memory
• ai-agents
• amazon-bedrock
• amazon-web-services
• autonomous-agents
• aws-lambda
• civic-tech
• cockroachdb
• community
• distributed-sql
• docker
• embeddings
• fastapi
• mcp
• model-context-protocol
• multi-agent
• mutual-aid
• pytest
• python
• semantic-search
• sql
• strands-agents
• titan-embeddings
• vector-database
• vector-search