Custodian

Icon
Privacy
Tools
HUD
Honestly
Recalls
Main Page
Welcome

Inspiration

AI assistants are brilliant for five minutes and forgetful forever. Every new chat starts from zero, so you re-explain who you are, what you're working on and what you already decided. Worse, they say things that didn't happen: "I've added that to your calendar" when the call was blocked, or a "memory" of something you never said that then comes back as fact in every later chat. And to get any of it, your whole life has to live on someone else's server.

We wanted the opposite: an assistant you host yourself, that remembers everything you've told it, that can't claim to have done something it didn't, and that you can talk to like Jarvis to run your PC.

What it does

Custodian is a private, single-user AI assistant:

Lifetime memory. Every chat, even a brand-new one, gets the facts it has saved about you and whatever you said in earlier conversations that bears on your message. Mention your sister's flight in passing on Monday, ask "when is my sister arriving?" in a new chat on Friday, and it knows.
Accuracy checks. Every reply is compared with what the turn actually did (which tool calls succeeded, which were blocked, what was saved) before you see it. A false claim gets one repair round, and if the rewrite still fails, a visible note says what really happened.
Permissions you control. Calendar, notes, Gmail, Drive, files, web and your PC each have their own switch, all off until you turn them on. Every tool call goes through a permission gate and runs in a locked-down sandbox process.
Jarvis for Windows. Say "Hey Jarvis" to open apps, play music, set the volume, and hear spoken replies, with a heads-up display of your system and your day.
Privacy by design. Messages, memories and embeddings are encrypted at rest. A Where your data goes page shows every service involved. Secrets are withheld from cloud models, and you can export or erase everything.

How we built it

Model: NVIDIA Nemotron on Nebius Token Factory, or our own fine-tuned custodian-nemotron running locally in Ollama.
Backend: FastAPI with one agent loop for every chat and scheduled job: skill selection, then memory retrieval, then model, then permission gate and sandbox, then accuracy check.
Storage: SQLite with FTS5, with every private column Fernet-encrypted. Vector search is NumPy over decrypted embeddings held in RAM.
App: one Expo (React Native) codebase for iOS, Android and the web.
Desktop: a Python tray app with openWakeWord, Whisper speech-to-text on the local GPU, and Piper text-to-speech.

Hybrid recall. Keyword search (BM25) and meaning search (cosine similarity over nomic-embed-text embeddings) produce two rankings, which we merge with reciprocal rank fusion:

$$ \text{score}(d) = \sum_{r \in {\text{BM25},\ \text{cosine}}} \frac{1}{k + \text{rank}_r(d)}, \qquad k = 60 $$

A raw cosine score isn't enough on its own: with nomic-embed-text, a message like "hello" still scores about $0.5$ against almost anything. So a meaning match must also stand out from everything else that query was compared with:

$$ s(q, d) \ge s_{\min} \quad\text{and}\quad s(q, d) \ge \mu_q + z\,\sigma_q $$

where $\mu_q$ and $\sigma_q$ are the mean and standard deviation of the query's similarities. On our paraphrase eval, keywords alone found $0/10$ facts worded differently from how they were said; hybrid search found $10/10$.

Grounding, not a judge model. The accuracy checks are deterministic rules (grounding.py). Every number and name in a saved memory must appear in something you actually said, and every claim in a reply must match the turn's record of what happened. They're cheap, predictable, and biased toward never flagging a true statement. In our eval they caught $24/24$ scripted false claims with $0/17$ true replies wrongly flagged.

Fine-tuning. We generated training conversations by running scenarios through the real agent loop with a scripted teacher, kept only the ones our own accuracy checks passed unchanged, and trained LoRA adapters on Nemotron. Each round is compared against the current model on 1,000 everyday requests and held-out conversations before we switch.

Challenges we ran into

Small models lie confidently. A 4B model says "Done!" when a permission was denied, which is why the grounding layer exists. Getting it right meant handling cases like a song title ending in "?" ("What We Cookin?"): the checker read "Playing 'What We Cookin?'" as a question rather than a claim, so the model replied with that line to everything and nothing caught it.
Better on average isn't better. Fine-tune v6 scored higher overall ($95.4\%$ vs $94.4\%$) but got worse at questions about uploaded files ($4/6 \to 1/6$). We kept v4. Per-behaviour evals mattered more than the headline number.
The model's answer disappeared. Through Ollama's OpenAI-compatible API, our local model sometimes put its entire final answer in the hidden reasoning field and left the visible content empty. Turning reasoning off for that endpoint fixed it, and the app now falls back gracefully instead of showing nothing.
Windows is full of surprises. WhatsApp is a Store app with no .exe to launch, the wake word had to tell "Hey Jarvis" from the TV, and a frameless always-on-top window has to be told explicitly to hide when you press ✕.
Memory you can trust. Saving too little makes the assistant forgetful; saving too much makes it invent a past. Requiring memories to be in your own words, and never recalling secrets or off-the-record chats, took many iterations.

What we learned

Verify instead of trusting the prompt. Checking the model's work against the record caught failures that no amount of prompt engineering fixed.
Retrieval quality is mostly about what you leave out: small talk, the assistant's own past mistakes, secrets.
Evals are product features. Every number above is regenerated by a script, never typed by hand, which let us change things quickly without guessing.
Privacy is a UX problem as much as an encryption problem. People trust what they can see, so every reply shows what shaped it and every setting shows where data goes.

What's next

Group sharing for families, a mobile wake word, and Custodian Cloud for people who want a hosted instance that still keeps their data encrypted with their own key.