Membrane

Your AI memory update might help one agent and quietly break another. We catch it before customers do

Comment

Membrane

Inspiration

SurrealDB calls itself "the context and memory layer for AI agents." This problem lives right there. If an agent relies on memory, one bad memory update can quietly change what the agent does. Membrane tests that memory layer before a change reaches customers, so the memory SurrealDB stores can be trusted.

What it does

AI agents share memories, but the same memory can help one agent and hurt another. Membrane tests how any shared memory changes each agent's decisions. It flags where the memory makes performance worse and shows exactly what changed. It's like crash-testing memory before it spreads across your AI workforce.

Here is how a test works:

  • The same ticket goes to the same agent twice: once without memory (control) and once with memory added (treatment).
  • Each arm runs five times.
  • A deterministic grader checks the agent's structured decision (decision, linked issue, symptom class, severity) against an answer key. No LLM judge is involved.
  • Each case gets a verdict: helped, harmed (correct before, wrong after), no change, or inconclusive.
  • Every verdict points to the exact fields that were graded, and every run links to its full trace.

Any team can plug in its own memory, agent, and answer key. Our demo uses one candidate memory to show the workflow.

How we built it

  • Code: a Bun and TypeScript monorepo, with a Next.js impact screen, a Hono API, and an eval runner.
  • Memory storage: SurrealDB keeps memory candidates as immutable versions, so every change can be traced.
  • Ticket data: an Eon connector (OAuth client-credentials) pulls support tickets from synthetic companies built on Era.
  • Traces: Arize Phoenix stores the datasets, the paired control and treatment experiments, and span-level traces with token counts for every call.
  • Pluggable agents: the harness runs any agent, with the same prompt in both arms. Memory is added as a fixed text block.

Challenges we ran into

  • Memory excluded its own evidence. The first memory said "failures reported since early October 2026," but our tickets were from 2024 and 2025. The memory ruled out the ticket it pointed to. We removed the date clause.
  • Subject and body disagreed. Some tickets have a subject that points one way and a body that points another. Haiku mostly followed the body.
  • A case that graded itself. One case was graded on an ID that only appeared in the memory. Control could never pass, and treatment could never fail. We fixed it.
  • Baseline already wrong. Four cases were inconclusive because the control arm already got them wrong. Their answer keys need review.
  • Environments overwrote each other. We generated multiple companies on Eon. When we generated the B2B ecommerce company a second time, it overwrote the previous environment.

Accomplishments that we're proud of

  • We generated B2B and B2C ecommerce companies on Eon, with 13 connectors each, and analyzed a large volume of data to find the issues hiding in it.
  • We ran a full end-to-end test on connector data: 230 paired calls (23 cases, 2 arms, 5 reps each), with traces in Phoenix.
  • We caught a real regression. On ticket 4005880, a persistence issue, the triage agent went from 5/5 correct without memory to 1/5 with it. The memory also led it to link the wrong known issue (4000197). This is the kind of flip Membrane is built to catch.
  • We report the negative results too. The memory linked the correct known issue in none of the 8 HELP cases. The support agent had no regressions and no improvement.

What we learned

  • One memory, two agents, two outcomes. Support was unchanged. Triage flipped on one case. Every agent needs its own test.
  • Memory is code. Its wording, date window, and missing details all change behavior.
  • Cost depends on the case. In one run, memory made the model reason about 16x harder. In another, it cut completion tokens by 13% by handing over the answer. Measure cost per successful case.
  • Copying is not judgment. If the answer only appears in the memory, you are measuring copying.
  • Answer keys must be machine-readable. Prose pass/fail columns fail silently. Every expected value should be a structured field with a fixed vocabulary, exported as CSV.
  • Deterministic grading is easier to defend. Every verdict traces back to one field.

Limits of this run: one model, one memory candidate, 23 cases from one simulated company, and 5 reps per arm. The triage flip rests on one ticket. A parallel run on a different snapshot showed no flip. We need more cases before drawing broad conclusions.

What's next for Membrane

Membrane will close the loop from finding a problem to fixing it. A person approves every change.

  1. Optimize the evals. Grow the golden set from 23 cases to 100 or more, across more tickets and both the B2B and B2C companies. Keep a held-out test set that the optimizer never sees. Have a second person label a sample, and resolve the four inconclusive answer keys.
  2. Recommend memory changes. When a memory harms an agent, Membrane finds the part that caused the flip (for example, the known-issue link on E8) and proposes a specific edit. Each edit runs the same paired A/B test on held-out cases before anyone sees it.
  3. Human decision. The impact screen shows the proposed edit, the before and after verdicts, and the traces side by side. A person decides whether to ship it.
  4. Test a symptom-describing memory. Run it as a labeled experiment on the same cases. No HELP case has passed yet, so this tells us whether helping an agent is possible at all.

The optimizer never sees the held-out cases, so a memory can't be tuned to pass the exact test it's graded on.

Built With

Share this project:

Updates

Submission history