Selective Memory

Inspiration Every AI I use (Claude, ChatGPT, Cursor) keeps its own hidden memory of me. I can't see what each one knows, I can't stop one from knowing something sensitive, and I repeat myself in every app. I wanted one memory that I own, where I decide what each AI is allowed to know.

What it does

Selective Memory is a personal memory hub that AI apps connect to through MCP (Model Context Protocol).

• One place for your data: see everything AIs know about you, plus "What Claude has" and "What ChatGPT has" views.
• Consent per memory: every fact is Open to AIs, Choose AIs (pick which apps), or Sensitive (no AI sees it).
• Asks before keeping sensitive things: health, money and job-search facts wait for your approval. Card numbers and passwords are used once and never stored.
• Claude hands over its memory: Claude imports what it knows into the hub, its built-in memory is switched off, and from then on it reads and saves only through Selective Memory.
• Every read is logged: see which AI read what, and when.
• Deletion you can prove: real deletes, a fingerprint-only receipt, and a search that confirms 0 matches.
• A key per app: disconnect one AI or make it read-only without touching the others.

Demo: I tell Claude I'm vegetarian and love Japanese food, then ask for dinner ideas and get vegetarian Japanese places. I mark "Japanese food" Sensitive, open a new chat and ask again, and Claude no longer knows it.

How we built it

• MCP server (Node, official MCP SDK): stdio for Claude Desktop, Streamable HTTP for ChatGPT, Cursor, VS Code and claude.ai. Tools: recall_memories, remember_fact, forget_memory, import_memories.
• Backend: FastAPI + SQLite, per-app tokens stored as SHA-256 hashes, per-memory access rules, activity log.
• Claude API extracts facts from chat with strict tool-use schemas. A deterministic privacy policy decides auto-save / ask / use once / ignore, and learns from your answers.
• Frontend: React + TypeScript + Vite + Tailwind.
• 37 automated tests.

Challenges

• Getting an AI to rely only on our memory: we imported its memory, turned off its built-in memory, and gave it a standing instruction to check the hub.
• Connecting web AIs to a local server: we used a secure tunnel with a per-app token in the URL.
• Designing privacy rules that are predictable rather than left to the model's guess.

What we learned

MCP makes memory portable across AIs. The hard part isn't storing memories, it's consent: giving people simple, visible control over what each AI can know.

What's next

A hosted hub with sign-in so web AIs connect in one click, encrypted sync across devices, and an SDK so agent builders get consent-aware memory built in.

Built With

• anthropic
• claude
• fastapi
• mcp
• node.js
• python
• react
• sqlite
• tailwindcss
• typescript
• vite