Selfware

Inspiration

Your inbox already knows the answers: the budget someone confirmed, the date a meeting moved, who owns a project. Finding them is the hard part. We wanted an assistant that answers questions about your own email and shows the emails behind every answer.

Many people organize their notes with PARA (Projects, Areas, Resources, Archive). That gave us a question worth testing, not just a feature to build:

Does organizing a mailbox into PARA folders make retrieval better than searching the whole mailbox at once?

What it does

Selfware turns your mailbox into a second brain you can ask in plain language.

Organizes your email. Each new email is classified into a PARA folder and indexed.
Answers with sources. You ask a question, and Selfware answers it with the emails it is based on: sender, date and folder.
Searches two ways. A toggle lets you search the whole mailbox (flat) or only the PARA folder chosen for your question, and compare the results.
Says "I don't know." When the evidence isn't there, Selfware abstains instead of making something up. Its confidence comes from the answering model's token log-probabilities, and the threshold lives in code, not in the prompt.
Tracks what it costs. A cost panel shows the dollars, tokens and latency of each model as you use the app.

How we built it

Selfware is a Python 3.14 monorepo, split among our four team members into four parts: email ingestion, retrieval, the answering agent with its evaluation, and the API with the dashboard. Every model runs on Nebius Token Factory through a single client, core/llm.py:

NVIDIA Nemotron Nano classifies each incoming email into a PARA folder. It is cheap enough to run on every message.
Qwen3-Embedding-8B embeds emails and questions into a Chroma index.
NVIDIA Nemotron Super picks the PARA folder to search for each question.
NVIDIA Nemotron Ultra writes the answer, and its token log-probabilities give the confidence score.
Qwen3-235B, from a different model family than the one that answers, served as an independent RAGAS judge in our full evaluation runs.

A FastAPI backend serves a React dashboard, and OpenTelemetry records every model call with its tokens, latency and cost. A mock mode runs the whole app without API keys, so our CI never spends credits.

Evaluating it honestly

We built the evaluation before we trusted any demo. We used the public Enron corpus: two real mailboxes, those of Vince Kaminski (research, 3,084 emails) and Richard Shapiro (government affairs, 1,179 emails). Each email's PARA label comes from the folder its owner filed it in, and we fixed that mapping before writing any question.

We wrote 250 questions, 125 per mailbox: 100 answerable ones, each checked by hand against its source email, and 25 whose answer is not in the mailbox. So that the questions would not copy the emails' wording, we wrote each one in two steps: first we extracted a fact from the email, then we wrote the question from that fact alone.

For each question $q$ with evidence email $e_q$:

$$ \text{hit@}k = \frac{1}{|Q|}\sum_{q \in Q} \mathbf{1}\left[\operatorname{rank}(e_q) \le k\right], \qquad \text{MRR} = \frac{1}{|Q|}\sum_{q \in Q} \frac{\mathbf{1}\left[\operatorname{rank}(e_q) \le k\right]}{\operatorname{rank}(e_q)} $$

with $k = 5$. For each question we compute $\Delta = \text{PARA} - \text{flat}$ and report its mean with a 95% paired bootstrap interval. An oracle arm sends each question to the folder where the owner filed the answer. This separates two questions: does the right folder help, and can a model find it?

hit@5	Flat	PARA, owner's folder	PARA, routed by Super	PARA, routed by Ultra
Kaminski	0.91	0.94	0.48	0.40
Shapiro	0.95	0.96	0.40	0.38

Challenges we ran into

Picking the folder is the hard part. Given a one-line question, both Nemotron Super and Nemotron Ultra send most questions to Resources (73–94%). A question reads like a request for information, while people file the answer under the project or area it belongs to. As a result, flat search still finds more than the PARA filter today.
Reasoning models count their reasoning toward max_tokens. With a cap of 4 tokens, the classifier received content=None, because the model was still reasoning, and it crashed. Once we fixed that, a closer look showed that the parser worked and that the bias toward Resources came from how the prompt frames the question.
Pinned dependencies. RAGAS, LangChain and the OpenAI 3.x SDK only work together on specific versions, and some incompatibilities only appear at import time.
Real people's email. The Enron corpus is public, but most of it belongs to people who had nothing to do with the case. We publish only the questions and the folder mappings, never the emails.
A limited budget. Every run that spends credits starts with a cost estimate, and our integration tests report the tokens they consume.

Accomplishments that we're proud of

An evaluation that answers our question. When a question goes to the owner's folder, PARA ranks the answer higher in both mailboxes: MRR rises by $+0.05\ [+0.02, +0.09]$ and $+0.02\ [+0.01, +0.04]$. We can show where PARA adds value and what stands in the way.
250 questions over two real mailboxes, with all 200 answerable ones checked by hand against their source email. The same pattern appears in both mailboxes.
An assistant that knows when it doesn't know, with every answer traceable to its emails.
Every model call is measured. The app's cost panel prices each call, and the whole evaluation cost about US\$6 in Token Factory credits.
One client for every model. Nano, Super, Ultra, the embedder and the judge all go through core/llm.py.

What we learned

Split the hypothesis into parts you can measure. "PARA doesn't help" and "the folder helps, but the router can't find it" are different findings, and the oracle arm told them apart.
Routing depends on the user, not on model size. A model four times bigger leaned the same way. What the router needs to know, how this person files things, is not in the question.
The ceiling matters. Flat search already finds 91–95% of the answers in a mailbox of a few thousand emails. That leaves PARA little room in hit@5, so its gain shows up in the ranking.
Check the questions by hand. Our human review caught AI-drafted questions that copied the email's wording or asked for more than one fact, and we rewrote them.

What's next for Selfware

Let the user's context pick the folder. The results point in one direction: a folder adds value when it reflects how the user actually works. The project you are working on, or a folder you choose, should guide the search, rather than a guess based on the question.
Test on bigger mailboxes. With tens of thousands of emails, flat search has much more to sift through, so the right folder should have more room to help.
A live PARA tree. The dashboard's folder view still shows sample data; the next step is to serve it from the real index.