Inspiration
A simple request like “take a picture and email it to me” involves opening apps, capturing a photo, finding the file, attaching it, and sending it.
We wanted to make those workflows easier through natural language. As an agent gains control over a computer, users also need to understand its actions and decide when it can proceed. That shaped CommandOS around three priorities: speed, user control, and transparency.
What it does
CommandOS is a voice-controlled AI agent for macOS. Users give it a spoken or typed instruction, and it coordinates tools and applications to carry out the task.
Its capabilities include file organization, application control, browser interactions, photo capture, and Gmail preparation. Our GitHub demonstration creates an empty temporary repository, requests approval during consequential steps, and verifies its deletion.
The dashboard makes execution visible. Users can inspect each step’s arguments, permission decision, output, and verification checks. Sending an email or deleting a repository requires approval.
CommandOS also includes Safety Lab, which runs controlled evaluations of behaviors such as respecting declined approvals, handling hostile instructions embedded in content, and recovering from changed UI controls.
How we built it
We built the desktop interface with Electron and TypeScript, supported by a native Swift helper that interacts with macOS through Accessibility APIs, Apple Events, and system services.
The execution flow is:
Voice or text → routing → workflow or AI plan → permission checks → execution → verification
Recognized commands can use predefined workflows, reducing the need for model planning. More complex requests use an AI planner to construct tasks with dependencies.
The runtime schedules those tasks, manages shared resources such as keyboard focus, handles approvals, and checks results. OpenAI models support planning, while ElevenLabs supports voice functionality. SurrealDB Agent Memory can store run outcomes and verified procedures.
We used AI coding assistance during implementation, debugging, and test development.
Challenges we ran into
Coordinating applications: Opening Messages while Photo Booth prepared a capture stole keyboard focus. We added resource locks and clearer task dependencies so foreground actions execute in sequence.
Handling changing interfaces: Accessibility references can become stale as pages update. We worked on finding controls again through their labels and checking their current state before interacting. Verifying completion: Gmail attachments and browser redirects introduced uncertainty. We added checks for observable results, including attachment
Built With
- anchor-browser
- arize
- daytona
- electron
- elevenlabs
- macos
- node.js
- openai
- opentelemetry
- playwright
- react
- surrealdb
- swift
- typescript
- vite
- zod
