PrivateDesk

PrivateDesk MemoryAgent gives every AI principal its own private, persistent memory — with isolation that's structural and provable by a test, not promised. Runs on Qwen, cloud or fully local.

Comment

Inspiration

Every serious AI assistant now has "memory" — but almost none of it is private by construction. On the big platforms, multi-tenant isolation is your job on every query: one missing metadata filter and Client A's memory leaks into Client B's answer.

I wanted to build memory for a setting where that leak isn't a bug ticket — it's a career-ending event. In a law firm, the wall between two matters is the profession's ethical wall. In a clinic, it's patient confidentiality. So the question became: can I make per-principal memory where isolation is structural and provable, not a promise you re-earn on every request? PrivateDesk MemoryAgent is my answer — the same engine runs a legal "ethical wall" demo and a healthcare "confidentiality" demo, because swapping domains is data, not code.

What it does

Every AI principal gets its own private, persistent, isolated memory that demonstrates the four MemoryAgent behaviors:

  • Accumulation — facts from a conversation are distilled into durable memories (PostgreSQL + a Qdrant vector store) and recalled, unprompted, in later sessions.
  • Isolation — each principal has an opaque, server-minted namespace (ns_ + 16 random bytes). All vector retrieval flows through a single chokepoint that is always namespace-scoped, so one assistant physically cannot read another's memory. A guard test fails the build if anything crosses the wall.
  • Forgetting — superseding facts are auto-detected and the old ones retired; an expiry + low-salience sweep prunes stale memory on demand.
  • Bounded recall — a reranker injects only the top few de-duplicated memories under a token budget.
  • Human-in-the-loop — the assistant drafts actions; a human approves before anything is "done."

How I built it

A FastAPI backend (the memory engine, a LiteLLM provider seam, and the Qdrant isolation chokepoint) serves a Next.js "cockpit" that visualizes the memory store, the live recall trace, the audit log, and the ethical wall side by side.

The write path is extract → embed → dedup/supersede → persist → audit. Extraction and the supersession verdict use Qwen qwen-plus with structured JSON output; embeddings use text-embedding-v4 (1024-d, multilingual); marquee turns use flagship qwen3-max.

Bounded recall is a small, legible reranker rather than raw cosine. Each candidate is scored:

$$ \text{score} = w_{\text{sim}}\cdot s_{\cos} \;+\; w_{\text{sal}}\cdot \sigma \;+\; w_{\text{rec}}\cdot r, \qquad (w_{\text{sim}}, w_{\text{sal}}, w_{\text{rec}}) = (0.6,\, 0.25,\, 0.15) $$

where $s_{\cos}$ is cosine similarity, $\sigma$ is the memory's salience, and recency decays with age:

$$ r(t) = e^{-\,\Delta t_{\text{days}}/30} $$

giving memory a half-life of a few weeks. I then take the top-$k$ ($k=6$ from $20$ candidates) with a greedy diversity filter — any candidate whose token-set Jaccard overlap with an already-chosen memory is $\ge 0.6$ is skipped, so a cluster of templated near-clones can't crowd out distinct context.

The whole thing is one env var away from being fully local: with DASHSCOPE_API_KEY set it runs on Qwen Cloud (DashScope); empty, it runs open-weight Qwen3 via Ollama. One terraform apply stands up the entire stack on Alibaba Cloud ECS (VPC, security group, EIP), ships the code, and seeds the demo.

Challenges I ran into

The half-indexed seed. The most instructive bug: seeding 100+ memories at once left recall returning partial results. Rows were in Postgres, but a burst of vector upserts while Qdrant was still warming up wasn't durable — some points silently never indexed. The fix was small and load-bearing: make every upsert wait=True so the write is durable before returning. Isolation is worthless if the store is quietly lossy.

Isolation beyond the obvious. Scoping the vector reader was step one. The subtler leak is the LLM prompt cache — a shared provider cache can become a cross-principal timing side-channel. I partition the cache per principal rather than crudely disabling it, keeping the speed-up without opening the wall.

Supersession vs. dedup. A high cosine cutoff misses updates that reword a fact ("$\$4.2\text{M}$ ceiling" → "raised to $\$5.0\text{M}$"). I scan the nearest neighbors above a lower floor and let qwen-plus classify each relationship as supersedes / duplicate / distinct — so a changed fact retires the old one instead of piling up a contradiction.

A humbler one: the Next.js build OOM-thrashed on a 4 GB box during deploy. I resized to a 2 vCPU / 8 GB instance (about half the original cost) and added a swapfile as a safety net — a reminder that "it works on my machine" and "it builds on the target" are different claims.

What I learned

Privacy for AI memory isn't about where the bytes live — it's isolation, open weights, and never training on your data, and Qwen makes all three possible at once. The most valuable design decision was funneling every vector read through one function so the security property could be stated in a single sentence and tested. When your invariant fits on one line, a guard test can guarantee it — and "provable" stops being a marketing word.

Built With

Share this project:

Updates