CommandOS

A voice agent that floats over your screen, finishes everyday Mac tasks in ~2 seconds, checks its own work, asks before anything risky, and can undo it.

Comment 1

Inspiration

AI agents can use computers now, but they're slow: watch the screen, think, click, repeat. We wanted something you can just talk to while you work, that does the small things you do 50 times a day in a couple of seconds, and that proves it actually worked.

What it does

CommandOS is a voice-driven agent for macOS. A see-through overlay floats above every app. Say "open Chrome and Gmail and email myself hi", "open Luma and figure out how many hackathons I have this month", or "take a photo of me and make it my wallpaper", and you watch it happen on your real screen:

  • Instant for everyday tasks: common commands run as fixed steps with no AI call at all. Routing takes milliseconds.
  • Uses AI only when needed: a fast model plans simple tasks, a stronger one handles harder ones, and a deep model is kept for genuinely ambiguous work. Steps start running while the model is still writing the plan.
  • Checks its own work: every step is verified against the real system (the file exists, the field holds the value, Gmail confirmed the send).
  • Safe by design: a policy engine, not the AI, decides what needs your approval (deleting, sending to others, payments). You can undo reversible changes, stop everything with one shortcut, and every action goes in a tamper-evident audit log.
  • Learns: a verified procedure is saved to memory, and "do the same for…" replays it without re-planning.

How we built it

  • Electron + React + Tailwind desktop app with a sandboxed renderer and validated IPC, and a Swift helper for native macOS control (Accessibility, Spotlight, on-device Vision OCR, ScreenCaptureKit, Keychain).
  • OpenAI GPT-6 models routed by difficulty (Luna / Sol / Astra), with streamed tool calls, hedged requests and Fast mode.
  • ElevenLabs Scribe v2 Realtime for speech-to-text.
  • Daytona sandboxes run and fix code safely: CommandOS stages a repo, runs its tests in an isolated sandbox with the network blocked, and has a patch proposed and re-tested there before you approve applying it.
  • Arize AX traces every route, plan, model call, tool, approval and verification step.
  • Anchor Browser cloud sessions for web tasks, and SurrealDB Agent Memory for learned procedures.

Results (measured live)

  • Everyday commands: 2.27 s median from prompt to done, 1.1 s to the first action (15 live runs, every output checked).
  • "How many hackathons do I have this month" on Luma: 7.9 s, read directly from the signed-in browser.
  • All six sponsor integrations verified with real API calls; six end-to-end demo workflows pass; 239 automated tests.

Challenges

GPT-6 returns one tool call per response, so we built a batch "schedule" tool that runs each step as soon as its JSON streams in. Chrome hides page contents from macOS Accessibility, so we turn on the screen-reader mode to read pages you're signed into. And keeping the overlay from stealing the keystrokes the agent sends to other apps took a non-activating macOS panel.

What we learned

The fastest AI call is the one you don't make. Deterministic code first, one fast model call second, deep reasoning last.

What's next

Background operation without taking over the screen, more apps, and a signed Mac app.

Built With

Share this project:

Updates

Submission history