Inspiration
Saathi (साथी) means friend in Nepali. That's what I wanted from a personal assistant: something that knows my week, helps without being asked twice, and can be trusted with the private parts of my life. My week runs across work shifts, study, client work and football, and most of it sits in my calendar and inbox. I didn't want to hand my email and contacts to a cloud model in plain text, or rely on a prompt to stop it sending an email I hadn't approved.
What it does
Say "Hey Saathi" and ask. It reads and drafts Gmail, manages Google Calendar, keeps tasks, reminders and timers, controls music and apps on the Mac, searches the web, reads the screen, remembers things about you and learns your routines. It answers out loud and puts lists, plans and briefings on a full-screen interface.
The home screen shows my next meeting, open tasks and the weather, with quick-action chips, a shelf of my saved skills and a one-tap model switch (Auto, Lightning, Super, Ultra). It steps aside while I talk, and replies fade on their own so the orb is left clean.
Meeting prep. "Prep me for my 2pm" finds the meeting, then pulls my recent email with each attendee, what Saathi remembers about them, and a Tavily search on their company, all in parallel. Nemotron 3 Ultra turns it into a briefing card and a short spoken summary.
Web search on Tavily. Search, news and page reads run on Tavily. The tool trace names it while it works, and the sources land on a card.
Reusable skills. Do something once, then say "save that as my weekly report skill". Saathi stores the steps and the exact tool calls it made, and the skill shows up on the shelf. Skills are plain Markdown files on the Mac, and Agent Skills style folders (name/SKILL.md) load too. Saathi only sees each skill's name and description until it runs one.
Voice. The demo's requests are spoken, not typed: wake-by-name, local speech recognition and the model run end to end. In the video, the requests and the voiceover are an AI clone of my voice (ElevenLabs) played into Saathi's microphone. Saathi's replies are live, in its own local voice.
Three design choices make it something I'd trust with my own accounts:
- Local senses. Wake-by-name, speech recognition (Whisper on the Mac's GPU) and memory search run on the Mac. No audio leaves the machine.
- A privacy vault. Email addresses, phone numbers and card numbers are replaced with placeholders like
[EMAIL_1]in everything sent to Nebius: saved memories, what I say, and tool results such as emails. Real values are restored on the Mac right before a tool runs and before Saathi speaks. - A confirmation gate in code. Sending email or texts, deleting anything, or editing a meeting with other attendees needs my spoken yes in the same turn. Arguments the model passes that start with an underscore are stripped, so it can't approve itself. Running a skill goes through the same gate.
How I built it
- Nemotron on Token Factory. A backend over Token Factory's OpenAI-compatible API offers all 63 tools as function tools. Every call still goes through one dispatcher and the confirmation gate, and Nemotron's
<think>output is stripped so it's never read aloud. - Routing across the Nemotron family. I probed every NVIDIA model on Token Factory for working tool calls, then split the work: Nemotron 3.5 Lightning for quick commands, Nemotron 3 Super 120B for everyday requests, Nemotron 3 Ultra 550B for planning and meeting prep. The router is rule-based, so it adds no latency and states why it picked a model. If Lightning or Ultra fails before a tool has run, the turn retries on Super. Once a tool has run it doesn't retry, so nothing happens twice.
- Vision through Nemotron 3.5 Super VL, which describes a screenshot for the text model.
- Tavily for web search (ranked results plus a short answer), news, page extraction and the company lookup in meeting prep, with DuckDuckGo as a fallback.
- A Brain panel showing which model answered, why, how long it took, the tokens used and how many values were masked.
- Wake-by-name. The speech detector finds short bursts of idle speech and local Whisper checks whether a burst starts with "Saathi". A trained wake-word model is used instead when one exists.
- Tests. 291 offline checks with a scripted fake model, covering routing, the vault end to end (the real phone number never appears in what's sent, yet the task saves it), fallback rules, meeting prep, Tavily, skills, wake-by-name, and a model that tries to send an email by setting its own confirmation flag. A separate eval runs everyday voice requests against the live models and scores whether the right tool was picked.
Built with Python, FastAPI, the OpenAI SDK against Nebius Token Factory, NVIDIA Nemotron, Tavily, Whisper (MLX), Silero VAD, Piper TTS, fastembed, React, Three.js, Electron, and the Gmail and Google Calendar APIs.
Challenges I ran into
- Model switching by voice looped. The first time I said "use Ultra", Nemotron called the switch tool eight times, because the tool description listed model names from another provider and spoken labels didn't match the IDs. The tool now lists the backend's own model families and accepts any words that identify one model.
- Keeping the vault useful. Saathi still has to act on masked data: a draft to
[EMAIL_1]must reach the real person, and dates like2026-10-28 18:00must not be mistaken for phone numbers. Arguments are unmasked just before the tool runs, the phone pattern rejects dates, and card numbers are only masked when they pass a Luhn check. - Speed for voice. Ultra gives the best plans but took 4 to 11 seconds a turn in my tests, far too slow for "set a timer". Routing keeps small requests on Lightning and saves Ultra for planning and meeting prep.
- No wake-word model for "Hey Saathi". Training one failed on my Python version, so Saathi listens for its name with local Whisper instead, and accepts the common ways Whisper writes it.
Accomplishments that I'm proud of
Saathi runs on four Nemotron models and shows which one answered each request and why. In a test turn, the phone number in "add a task to call 0412 345 678" was masked on the way to Nebius and still saved correctly in the task. When Nemotron tried to send an email, the gate asked, I said no, and nothing was sent.
What I learned
When I probed the five NVIDIA models on Token Factory, every one passed a simple tool-call check, but response times ranged from 0.6 to 3.1 seconds. For a voice assistant that gap matters more than a small difference in quality, which is why routing beat always using the biggest model. I also learned to tell the model outright that tool results are data, never instructions, because any email Saathi reads could have been written by anyone.
What's next for Saathi
- Run more of the reasoning on the Mac as smaller Nemotron models allow, so even less leaves the machine.
- Skills that trigger themselves: the morning kickoff when I first sit down, a follow-up draft after each meeting.
- Masking for names and street addresses, not only emails, phones and cards.
- A learned router trained on my own logged turns.
- Shorter spoken replies when a card is on screen.
Code (open source, MIT): github.com/Yogesh1533/saathi
Built by Yogesh Dhakal.
Log in or sign up for Devpost to join the conversation.