Cosmos

At your service. The dev agent that debugs itself. COSMOS reads its own error logs, patches its own source code, have watchers, controls your mac, runs its own tests, and restarts itself live.

Comment

Inspiration

I wanted an assistant that actually does things, not one that hands me a snippet and wishes me luck. Something that lives on my own machine, understands my Mac the way I do, and can take a goal and run with it: open the browser, dig through files, run the commands, watch the screen, remember what happened last time. A real second pair of hands.

The more capable it got, the more one idea nagged at me. If an agent can read logs and write code, why does it keep making me fix its own mistakes? So alongside the everyday work, I gave COSMOS the ability to repair itself. That became one feature among many, but the whole project is about an agent that can genuinely operate a computer on your behalf.

What it does

COSMOS is a locally hosted developer agent that runs as a webapp and drives your Mac end to end. You talk to it by voice or text, and it works across roughly 50 tools. Every capability has its own panel:

Agent. The core loop: observe, think, act. Shell commands, file read and write, git and GitHub, browser control through Chrome's debug protocol, screenshots with vision analysis, web search, document and PDF reading, and full macOS control through AppleScript, clicks and keystrokes. Give it a multi-step goal and it plans, executes, and self-verifies, streaming its thinking as it goes.

Kinesis. Record, understand, replay. It captures a UI workflow, uses the model to understand what each step meant rather than memorizing pixels, and replays it later even after the layout shifts. Macros stay editable in plain language.

Vision. Screen watchers. Pin a region of any webpage or your own screen and COSMOS polls it, tells you the moment it changes, and can fire a reflex action in response. An interactive preview browser lets you draw the region by hand.

Mutate. The self-healing panel. It reads its own audit trail and run traces, diagnoses its own failures, and patches its own source code behind real test gates, then restarts itself in place. You can also just tell it what to change about itself, frontend included.

Panel. A multi-agent swarm board. Spawn parallel workers on a task list, give each a persona, watch them run side by side, and broadcast instructions to the group.

Nexus. A live mind map of everything COSMOS knows: systems, skills, macros, people and running threads, as an explorable graph.

Dossier. Per-person files built from your communications. Who owes you what, what you promised, and the context behind each thread, refreshed daily.

Skills. Markdown playbooks injected into the agent's prompt. COSMOS notices repeated work and offers to save it as a reusable skill, all editable in-app.

Slack, Connectors, Memory. Drive it remotely from a Slack channel over Socket Mode. Plug in Google Workspace, ElevenLabs voice, or any MCP server. And it keeps long-term memory, lessons from failed runs, and semantic recall over past sessions.

Underneath: model fallback chains, a risk gate that confirms before anything destructive, an append-only audit log, undo snapshots, and a background scheduler.

How we built it

FastAPI backend on localhost, hand-built React HUD, GPT-5.6 for heavy reasoning and gpt-5.6-mini for fast reflexes like routing and summaries. The agent loop is the heart of it: about 50 tools wired into an observe-think-act cycle with a risk gate and a self-verify critic, streaming over websockets to the HUD.

I built it in close collaboration with Codex. I would describe the behavior or invariant I needed, let Codex draft the implementation, then interrogate it until we both trusted it. Codex was fastest at the mechanical middle of the work: translation layers, test scaffolding, and refactors that swept across dozens of files at once. The architecture decisions stayed human.

Challenges we ran into

Driving a real machine reliably. Controlling a Mac through AppleScript, clicks and the Chrome debug protocol means fighting timing, focus and permissions constantly. macOS permissions belong to the app that launched you, and I learned that the hard way when the camera silently refused every photo. An orphaned browser process once held a profile lock for two days and quietly broke Vision.

Making replay survive change. Kinesis had to replay workflows even after a page moved things around, which pushed me from recording coordinates to recording intent.

Letting the agent touch its own source safely. For Mutate, the scariest failure is a confident wrong patch, so nothing is trusted: per-file backups, a boot check in a disposable subprocess, tests as the judge, and restart via os.execv so the process keeps its PID and permissions while the UI reconnects on its own.

Swapping the brain mid-project. Moving to the OpenAI API without touching a 3,600-line agent loop meant a translation layer at the wire for tool calls, images, streaming deltas and stop reasons.

Accomplishments that we're proud of

An agent that genuinely operates a computer, not a chat window. It opens apps, runs real commands, reads the screen, replays workflows, and coordinates parallel sub-agents, all locally and all under a confirmation gate for anything risky.

The standout moment is COSMOS patching its own code and restarting itself in about 28 seconds with the same process ID and no lost UI session. Beyond that: 699 tests passing with zero failures, a full model swap to GPT-5.6 the agent loop never noticed, and features like Kinesis and Vision that hold up on messy real-world pages.

What we learned

Capability is easy to demo and hard to make reliable. The interesting problems were rarely the AI. They lived in process semantics, permission models, file locks and timing. GPT-5.6 was genuinely strong at reasoning over real evidence, whether that was planning a multi-step task or writing a minimal correct patch. And I learned that the way to trust a model with dangerous work is not a cleverer prompt, it is better gates around it.

What's next for Cosmos

Deeper autonomy on long-running goals, a native menu-bar shell with on-device speech recognition, richer Kinesis macros, and proactive suggestions that surface before you ask. Mutate gets a real timeline so you can watch the agent evolve week by week, and a shared lesson library so a fix one COSMOS learns can help every other install. It already runs your machine. Next it gets better at deciding what to do on its own.

Built With

Share this project:

Updates