Private Personal Assistant with Persistent Memory

An always-on personal assistant that retains long-term memory across chat sessions, learns custom procedural skills, and keeps user data fully local and encrypted.

Comment

Inspiration

AI assistants force a bad choice: they forget everything between sessions, or they remember everything inside a black box you can't see into. We wanted a personal assistant that becomes more useful the longer you use it, where you decide exactly what it knows about you.

What it does

Private Personal Assistant is a private, always-on assistant with long-term memory you can inspect.

  • Three memory stores: facts (preferences and background), episodes (summaries of past conversations) and skills (reusable workflows, such as a "Weekly Planning Routine").
  • Visible recall: every reply shows which memories it used.
  • Privacy control panel: search, edit, export or permanently delete any memory. A delete removes the stored text and its embedding together, and an automated test checks that the deleted memory can no longer be retrieved.
  • Skills: say "save this as a skill" and the workflow is stored and can be run again later.
  • Tools with consent: notes and calendar changes are proposed first and only happen after you confirm.
  • Dynamic model routing: a small Nemotron model handles everyday turns, and complex planning escalates to Nemotron-3-Ultra. The interface shows which model answered and a running cost counter.

How we built it

  • Models: NVIDIA Nemotron on Nebius Token Factory, through its OpenAI-compatible API. A small Nemotron model handles fast turns and Nemotron-3-Ultra-550b-a55b handles planning.
  • Embeddings: Qwen3-Embedding-8B on Token Factory, truncated to 1024 dimensions.
  • Backend: Python and FastAPI.
  • Memory: encrypted SQLite (SQLCipher). Embeddings live in the same database as the memories, so deleting a memory deletes its vector too.
  • Tools: notes and calendar behind a single tool interface with an allowlisted folder, timeouts, an audit log and a confirmation step for every write.
  • Interface: a minimalist web UI with a chat view and the privacy panel.

What Nebius and NVIDIA contribute: all reasoning runs on NVIDIA Nemotron models served by Nebius Token Factory. Routing between a small and a large Nemotron model keeps the assistant fast for everyday chat and affordable, because the expensive model is only used when a task needs real planning.

How a turn works: the message is embedded, the closest facts, episodes and skills are retrieved and placed in a clearly marked block of the prompt, and the router picks a model. After a session, a cheap extraction step turns the conversation into new facts and an episode summary. Near-duplicates update existing memories instead of piling up.

Challenges we ran into

  • Keeping memory clean: without deduplication, the same preference gets stored many times. We compare new facts with existing ones by embedding similarity and update instead of inserting.
  • Making deletion real: a vector index kept apart from the data can leave traces behind. Keeping embeddings in the same encrypted table made hard delete a single operation.
  • Encryption that can't silently turn off: the app is set to refuse to start when encryption is unavailable, so sensitive memory is never written to a plain database by accident.
  • Prompt injection through memory: content from tools or files could poison what the assistant remembers. We treat tool output as data, not instructions, and require confirmation before any change.
  • Setting up and deploying: getting billing, model IDs and persistent storage working took more time than expected.

Accomplishments that we're proud of

  • A test that deletes a memory and proves it can no longer be retrieved.
  • An assistant that remembers across separate sessions and shows what it used.
  • Users can see exactly what the assistant knows about them and remove it.
  • Cost-aware routing between two NVIDIA Nemotron models, with a visible cost counter.

What we learned

  • Extracting less and deduplicating more works better than storing everything.
  • Privacy claims need to be precise. Memory is encrypted at rest, and only retrieved snippets and text to be embedded reach the model provider. That is not the same as "everything stays local".
  • Small and large models complement each other: a cheap model is enough for most conversation, and the large one earns its cost on planning.
  • Memory should be treated as untrusted input, because anything the assistant reads can end up stored.

What's next for Private Personal Assistant with Persistent Memory

  • A Telegram interface, so the assistant reaches you where you already are.
  • Real email and calendar connectors with per-action approval.
  • A fully sandboxed tool runtime, for example with OpenShell or NemoClaw.
  • A local-embeddings option, so memory text never leaves the device.
  • Memory expiry and "forget after N days" rules.

Built With

Share this project:

Updates

Submission history