Soren

Talk to your Mac and get things done across apps, browsers, and Terminal without reaching for the keyboard.

Comment

Inspiration

I started Soren because I was tired of the gap between asking a computer for help and actually getting the work done.

Most assistants can explain how to do something, but then hand the task back to you. I wanted to say, “Open the latest project note,” “Run the tests,” or “Go to Storage settings,” and watch my Mac handle it—without hiding what it was doing.

What it does

Soren is a voice-first assistant for macOS.

Hold the talk shortcut, describe what you want, and Soren can work across native apps, browsers, Finder, and Terminal. It reads the current interface, finds the right controls, performs the action, and checks whether it worked.

A small notch interface shows what Soren is doing. When it needs to interact with something on screen, a guided cursor comes out of the notch and moves toward the target. Risky actions pause for approval, and every task can be stopped immediately.

Speech-to-text and text-to-speech run locally on the Mac. Soren only sends the context needed for reasoning to the selected model.

How I built it

The macOS app is written in Swift using SwiftUI, AppKit, Accessibility APIs, ScreenCaptureKit, and Core ML. It owns the microphone, local speech, windows, overlays, permissions, and all interaction with the operating system.

A Rust task engine turns each request into a typed task contract, validates the plan, tracks execution, and prevents unsafe or duplicate actions. OpenAI models handle language understanding and structured tool decisions, while the local engine remains responsible for what is actually allowed to happen.

The components communicate through JSON-RPC. Browser control uses a Chrome extension, WebSockets, and the Chrome DevTools Protocol. Native apps are controlled through their accessibility trees, with screenshot-based interaction used only when a reliable native target is unavailable.

GPT‑5.6 also helped me test Soren’s grounding layer. I compared macOS Accessibility trees with DOM geometry and screenshots, which exposed problems like duplicate labels, off-screen controls, and canvas elements. That’s why Soren uses Accessibility first and vision only when needed.

I ran through multiple experiments with screenshots and the DOM trees

Challenges

The hardest part was making actions dependable across very different Mac apps. Accessibility trees are often incomplete, duplicated, or structured differently between native and web-based applications.

Another challenge was recovery. If an action times out after being dispatched, blindly trying it again could send a message twice or repeat another important action. Soren keeps execution receipts and verifies the current state before deciding what to do next.

I also spent a surprising amount of time on motion. The notch, text, and guided cursor had to adapt to different MacBook and display sizes without feeling mechanical or getting cropped.

What I learned

I learned that useful computer control is less about generating clicks and more about grounding, evidence, and trust.

A fast model helps, but the experience only feels good when the native app can show what is happening, stop instantly, verify the result, and admit when it is unsure.

What's next

Next I want to expand the library of app-specific guides, improve long-running task support, and make it easier for people to teach Soren repeatable workflows of their own.

Built With

Share this project:

Updates