Memory Desk

Inspiration

Every AI conversation we'd ever had died the moment we closed the tab. Paste a document into ChatGPT on Monday, and by Friday you're pasting it again, because the model has no idea it ever saw it. That's not intelligence with amnesia — that's just amnesia with good vocabulary.

We also kept running into the opposite failure mode: assistants that, rather than admit they didn't know something, would confidently invent an answer that sounded like it came from your notes but didn't. We didn't want a bigger context window. We wanted a memory — something that persists between sessions, that you can point at and say "you actually said this on March 3rd," and that would rather say "I don't have that" than make something up.

CockroachDB's native VECTOR type was the piece that made this feel buildable rather than theoretical: a distributed SQL database that can hold your data and the geometry of its meaning in the same row, replicated and durable, with no separate vector store to keep in sync. That's where Memory Desk started — not "let's build a chatbot," but "let's build a database-backed memory an AI can actually trust."

What it does

Memory Desk is a personal memory layer that sits between you and an AI assistant. You feed it the things you want remembered — typed notes, screenshots, emails pulled straight from Gmail, documents pulled from Google Drive — and it embeds each one into a 3072-dimensional vector and stores it as a native VECTOR(3072) column in CockroachDB, namespaced to your own user table.

Later, you ask it a question in plain language. Memory Desk embeds the question, runs a nearest-neighbor search against your stored memories using CockroachDB's distributed vector index, pulls back the memories that are actually relevant, and hands only those to the model as grounding. The answer comes back with inline citations pointing at the exact memory each claim came from — click one, and you see the original note, screenshot, or email it was pulled from. If nothing in your memory answers the question, it says so instead of guessing.

Under the hood, similarity between a query vector $\mathbf{q}$ and a stored memory vector $\mathbf{m}$ is scored by cosine similarity,

$$ \text{sim}(\mathbf{q}, \mathbf{m}) = \frac{\mathbf{q} \cdot \mathbf{m}}{\lVert \mathbf{q} \rVert \, \lVert \mathbf{m} \rVert}, $$

and instead of brute-forcing that over every row, CockroachDB's distributed C-SPANN index partitions the vector space so a search touches a small, relevant slice of the table rather than the whole thing — the kind of approximate nearest-neighbor search that makes "search everything you've ever saved" feel instant instead of theoretical.

It ships two ways: a web frontend, and a native-feeling desktop app that wraps the same UI and points at a backend running on AWS EC2 behind an Elastic IP, so it's a real always-on service, not a localhost demo.

How we built it

The backend is a FastAPI service (api.py) that owns three jobs: ingest, embed, and retrieve. Ingestion normalizes wildly different inputs — plain text, screenshots that need OCR/vision before they're anything but pixels, Gmail messages, Drive documents — into clean text, chunks anything long enough to risk truncation, deduplicates near-identical content, and writes each chunk as its own row with its own embedding, tagged with metadata (source, timestamp, type) so citations can point back to something concrete.

Embeddings come from Google's gemini-embedding-001, chosen specifically because its 3072-dimension output matches the VECTOR(3072) column we defined in CockroachDB — we also wired up Hugging Face and Amazon Bedrock (Titan embeddings, Claude Haiku for chat, Textract for OCR) as swappable providers, since AWS services were part of the brief, but standardized on Gemini so every row in the table lives in the same vector space.

Storage and retrieval both live in CockroachDB Cloud. Each user gets their own logically namespaced table (MEMORY_TABLE, keyed by USER) so memories never bleed across accounts, and retrieval is a straight SQL query with a vector distance operator — no separate vector database, no sync job, no second point of failure.

On top of that sits a strict RAG prompt: the model is instructed to answer only from the retrieved memories, to cite the memory it drew each claim from, and to say it doesn't know rather than fill gaps. The frontend (MemoryDesk.jsx) surfaces that retrieval step visibly — you can see what was pulled back before you see the answer — because trusting a black box defeats the point of building a memory system in the first place.

The whole thing is deployed on an AWS EC2 instance with a static Elastic IP, and the desktop app is a thin native shell that just points its config at that same public endpoint, so the web app and the desktop app are the same product wearing two different windows.

Challenges we ran into

Dimension mismatches were a silent killer. Early on we tried treating embedding providers as interchangeable — until we learned the hard way that a 384-dimension Hugging Face embedding and a 3072-dimension Gemini embedding cannot coexist in the same VECTOR(3072) column. Every write from the "wrong" provider failed, and the fix wasn't clever code, it was discipline: pick one embedding space and enforce it everywhere.

Screenshots are not text. Getting image uploads to actually contribute usable memory — rather than just sitting in storage as an opaque blob — took real iteration: routing images through vision/OCR, cleaning up the extracted text enough that it embedded meaningfully, and making sure a screenshot of, say, a receipt or a chat log came back as a citable memory just like a typed note would.

Per-user isolation had to be real, not cosmetic. Namespacing memories per user sounds simple until you're deciding whether that lives at the table level, the row level, or the query level — we settled on per-user tables so a bug in a WHERE clause can never leak one person's memories into another's answer.

Keeping the model honest was harder than keeping it smart. It's easy to get an LLM to sound confident. Getting it to say "I don't have anything in your memory about that" instead of quietly synthesizing a plausible-sounding answer took real prompt iteration and testing against edge cases where the honest answer was "nothing."

Accomplishments that we're proud of

We're proud that Memory Desk's vectors aren't a bolt-on — they're first-class columns in a real, distributed, production-grade SQL database, which means memory here inherits CockroachDB's durability and replication for free instead of depending on a fragile side-store.

We're proud that every answer is inspectable: no claim in a Memory Desk response exists without a citation you can click through to the original memory, which is a genuinely harder product to build than one that just sounds confident.

And we're proud we shipped a real, always-on service — hosted on AWS, reachable from a browser or a desktop app — rather than a demo that only runs on the presenter's laptop.

What we learned

We learned that "grounded" is a design constraint, not a prompt tweak — it shapes ingestion, chunking, retrieval, and the prompt itself, all the way through. We learned that treating the database as the source of truth for both data and meaning (via native vector types) removes an entire category of synchronization bugs that a bolted-on vector store would have forced on us. And we learned, concretely, that embedding dimensionality is a contract — the moment you break it, you don't get a warning, you get silent failure at write time.

What's next for Memory Desk

• Memory that ages gracefully — surfacing what's stale, letting you correct or retire a memory instead of it living forever unchallenged.
• Proactive recall — instead of only answering when asked, quietly surfacing a relevant memory before you know you need it.
• More sources — Slack, calendar, browser history — anywhere a moment worth remembering already lives.
• Shared and team memory — multi-user tables where a team builds one collective, grounded memory instead of everyone starting from zero.
• Tighter CockroachDB integration — leaning further into distributed vector indexing as it matures, so retrieval stays fast as memory grows from hundreds of entries to hundreds of thousands.

Built With

• amazon-web-services
• cockroachdb
• gemini
• python