Inspiration

We were inspired by Clicky, an open-source Mac companion that explains things on your screen, pointing at buttons and walking you through software with graphics. We loved it, but kept thinking: explaining isn't always enough. Many people don't know where to go or how to do something, and they just want it done. So we asked: why not build a companion that actually does the task for you, safely?

Every family has one person who gets the messages: "Beta, OTP kahan daalna hai?", "This form won't submit", "How do I download my Aadhaar?". Government portals, bank sites and booking sites are a maze of menus, captchas and pop-ups, and our parents wait for us. Saathi ("companion") is for them.

What it does

  • Type it or say it. Press ⌘⇧Space and type, or hold Control + Option and speak: "download my Aadhaar", "open YouTube and play some good vibe music".
  • Saathi does it. It opens the right site, clicks, types and scrolls, while a marigold ring shows and narrates every step.
  • Safety is enforced in code:
    • It pauses before pay, submit, send, book or delete.
    • It never types passwords, OTPs, card or ID numbers. It hands those back with 🔒 "your turn".
    • Esc stops everything.
    • It clears cookie banners and pop-ups.
  • It learns. Successful tasks become skills. Replays find each button again on the live page, mostly with no model call, so it gets faster with use.
  • Receipts and routing. Every step is kept as a receipt with a screenshot, and every model call is logged with latency and cost, under a spend cap.

How we built it

  • App: native macOS, written in Swift with SwiftUI and AppKit, forked from Clicky (MIT).
  • Seeing the screen: Saathi reads real buttons and fields from the macOS Accessibility tree, with screenshots only when needed.
  • Privacy shield: on-device Apple Vision OCR blacks out Aadhaar, PAN, passport and card numbers, plus all password fields, before any screenshot leaves the Mac.
  • Thinking: NVIDIA Nemotron on Nebius Token Factory:
    • Nemotron 3.5 Super VL sees the screen and plans each step.
    • Nemotron 3 Nano makes the quick decisions: routing, skill matching, filling skill values.
  • Voice: push-to-talk with on-device Apple Speech.
  • Tavily finds the official steps and site, and warns you about lookalike domains.

Challenges we ran into

  • Big web apps hide suggestion lists deep in the accessibility tree, so we added keyboard-based suggestion picking and screen-change detection.
  • Speed: an early version made too many model calls. We rebuilt it around signals that need no model call: screen fingerprints, adaptive waits and one-step search.
  • Nemotron Nano reasons before it answers. Our first token cap cut it off mid-thought, so we tuned the allowance.
  • macOS permissions reset on every rebuild, which we fixed with a stable self-signed signing identity.

What we learned

For computer-use agents, reliability comes from good perception (the accessibility tree, change detection) and hard safety rules more than from bigger prompts. Saving successful runs as skills turns a careful agent into a fast one.

What's next

A guided "teach me" mode on the same engine, more Indian languages, family skill-sharing, Windows support, and verified skills for common government and bank portals.

Built With

  • accessibility-api
  • appkit
  • apple-speech
  • apple-vision
  • macos
  • nebius
  • nebius-token-factory
  • nemotron-nano
  • nemotron-super
  • nvidia-nemotron
  • openai-compatible-api
  • screencapturekit
  • swift
  • swiftpm
  • swiftui
  • tavily
Share this project:

Updates

Submission history