Adiutor

Logo

Inspiration

If you have RSI, a tremor, or arthritis, or you just can't use a mouse and keyboard for long, Windows already gives you Voice Access. It's reliable, but it works one command at a time. To email your boss a deck you say "Open Edge," "Click address bar," "Type outlook dot com," "Click New mail," "Click To," "Type Ahmed"… then "Show numbers," "Click 14," "Click Send." That's about 21 commands for one email, and you wait on every one.

We wanted the opposite: say the goal once, and let the computer do the clicking.

What it does

You say "Hey Adiutor, email my boss the Q3 deck and tell him I'm running ten minutes late." Then:

It hears you. A small island at the top of the screen turns amber while you talk and shows your words live. A 1.2-second pause mid-sentence doesn't end your turn, because slow or halting speech is the normal case, not an edge case.
It knows who "my boss" is. Memory is plain files on your laptop (USER.md, JOURNAL.md). Before each reply, Adiutor searches them locally and shows what it used: From memory · Ahmed Hassan · Q3 Review.pptx.
It does the work. It opens the browser, writes the email, attaches the right file, and fills every field.
It shows you before anything it can't undo. Nothing leaves until you say "send." The card shows To, Cc, Subject, the body, and the attachment. If anything changed mid-task, it's marked ("Mona Farouk · added"). Say "send" or press F9; say "cancel" or press F10. Enter never approves.
It looks things up without opening anything. "What's the best-rated coffee shop near me?" or "How much is this on Amazon?" is answered in the background from live web data.
It protects you from scams. Every link, download, and email recipient goes through a scam guard first. Lookalike sites and unverified installers are blocked until you approve them. "Find me the best deal" compares stores, coupons, and seller risk, and Adiutor stops at payment so you pay yourself.
It can build new skills by voice. "Make me a skill that…" hands the job to a coding agent that writes, tests, and installs the skill.
"Stop" always works. You can talk over it at any time.
It learns your routines. Do a task once while it watches (refill a prescription, say) and Adiutor writes the steps down as a plain-English SKILL.md, using button names rather than screen coordinates. Next time, "Refill lisinopril" runs the skill. You can edit it by voice or in Notepad.

One sentence and a yes.

How we built it

Voice: Cartesia. Speech-to-text and end-of-turn detection use ink-2 through Pipecat. The voice is sonic-3.6. Wake word detection ("Hey Adiutor") runs on the device with openWakeWord, so no audio leaves the laptop before it fires.
Brain: Claude (Anthropic). Claude plans the task, speaks every word, fills the values, writes the confirmation card, and writes and rewrites skills from one demonstration or one sentence.
Memory: Moss. A local Moss index over your people, places, files, past tasks, and skills gives Claude the few documents that matter each turn, instead of the whole history. The index is built and stored on the laptop and never pushed.
Hands: Windows-MCP. We built Adiutor on our own fork of Windows-MCP to drive the Windows desktop, with Playwright over Chrome DevTools for the browser. A guard blocks sends, submits, and deletes in the browser and shell unless you've just said yes.
Web data: Bright Data + Apify. A background WebData tool uses the Bright Data SERP API for search and the Bright Data Web Unlocker for pages that block bots. It also runs Apify Actors (Google Maps places, Amazon products) and the Apify RAG Web Browser for clean page text, so answers arrive without a browser window stealing focus.
Builder: Droid by Factory. "Make me a skill that…" runs Factory's droid CLI headless in the background. Droid writes, tests, and installs the skill, then Adiutor reads you a summary. Voice coding ("fix the failing test") runs at low autonomy: it never pushes or installs.
Browser: browser-use + CDP. Fast browser automation over the Chrome DevTools Protocol, with browser-use's tools mounted behind the same scam guard.
Judgment: Microsoft Decision-1. Through OpenRouter, Decision-1 gives a calibrated second opinion on gray-zone pages for the scam guard. It also scores how to route each request (act now, plan, answer, or ask) and whether a step actually succeeded.
Eyes: a fast vision model. When the accessibility tree can't find a target, a screenshot goes to a vision model that returns the point to click. We benchmarked 15+ models on real, crowded 1920×1080 Windows desktops (nine overlapping apps, 20 small targets such as tab close buttons and single grid cells). Claude Haiku 5.5 got 18–20 of 20 in about 1.2–2.2 s and we picked it. One surprise: overlaying a pixel coordinate grid on the screenshot never helped any model and often made them worse.
Face: the island. An Electron overlay in plain HTML and JS, designed in Figma first. Every state has its own color, and nothing on the island is smaller than 13 px, so you can read it from across the desk. It also has a reduced-motion mode.

Challenges we ran into

Turn-taking for slow speech. Most voice agents assume fast, fluent speakers and cut you off at the first pause. We tuned end-of-turn detection so pausing to think doesn't end your turn, while "stop" still cuts in instantly.
Safety without nagging. An agent that asks "are you sure?" every step is no better than Voice Access. We confirm only actions that can't be undone, show exactly what will happen, and enforce it in code for the browser and shell.
Finding the right click point. Accessibility trees are incomplete on real apps, and vision models disagree a lot. Several "fast" models were off by hundreds of pixels, and Gemini answers in a 0–1000 coordinate format rather than pixels. We built a benchmark harness so we could choose by measurement instead of by vibes.

Accomplishments that we're proud of

One sentence and a "yes" replaces a ~21-command Voice Access sequence for the same email.
A confirmation card that shows what changed instead of only asking "are you sure?"
Skills that are readable, editable plain English, written from one demonstration.
A clear, written statement of exactly what leaves the laptop, and when.

What we learned

Accessibility users aren't edge cases. Designing for halting speech and no hands made the agent better for everyone.
Measure before choosing. Our model benchmark overturned our assumptions twice: the grid overlay hurt accuracy, and the cheapest "flash" models weren't usable for clicking.
Use the right tool for each hop. Background web data from Bright Data and Apify beats driving a browser for "just tell me" questions. A coding agent (Droid) is the right way to write new skills.
Plain files beat hidden state. Memory and skills you can open in Notepad are easier to trust and easier to debug.

What's next for Adiutor

Guard native desktop clicks, not just the browser and shell.
Test with real users who have RSI and motor impairments, and run a proper study comparing it against Voice Access.
More built-in skills (calendar, files, forms), and sharing skills between people.
A healthcare-ready path: zero data retention and a BAA with our model providers.

Privacy: exactly what leaves your laptop

Stays on your laptop:

The wake word, detected on the device.
Memory: USER.md and JOURNAL.md.
Skills: the SKILL.md files.
Recordings and logs.
Memory search: the Moss index runs locally and is rebuilt from your files.

Sent to the cloud:

Cartesia: your speech after the wake word.
Anthropic: the conversation, screen text and screenshots, and memory.
Moss Cloud: the project key, a one-time model download, and usage telemetry. We never call push_index.
Bright Data / Apify: your web search queries and the URLs Adiutor reads for you.
Microsoft Decision-1 (via OpenRouter): for gray-zone pages only, the domain, the page title, and a short text excerpt. For routing, the request text and the option names.
Factory (Droid): only when you ask it to build a skill or code: the request and the files it works on.

A false wake sends that audio, plus the 1 second before it.