Avengents AI

Your AI coding agents are stuck at one desk. Avengents puts them in your pocket: pair a machine once, then run Claude Code, Cursor or Codex from your phone, by text or by voice.

Comment

Inspiration

AI coding agents live in a terminal on one machine. Walk away from that desk and you are blind: an agent may be waiting on a question you would answer in four seconds, or it may have finished twenty minutes ago and be sitting idle. The usual answer is to rent someone else's cloud and hand over your repository and your keys. We wanted the opposite. The agents stay on the computers you already own. What travels is you.

The project is also its own proof. Most of Avengents was written by the agents it controls, directed from a phone away from the desk.

What it does

Pair a machine once by scanning a QR code. After that every agent on it is one tap away: a real terminal with its session and its scrollback, not a chat transcript of one. Type, or hold the microphone and speak. Hand files in either direction. Open a project room where several people and several agents share one thread, so a teammate and a Claude Code session are the same kind of participant. Add friends, report and remove them. A Plus tier raises the daily voice-input allowance.

iPhone, iPad, Android, macOS and the browser all run the same app.

How we built it

The web app is React, TypeScript, Vite and Tailwind, with Zustand and TanStack Query for state and xterm.js for the terminal.

On your machine runs a small Node and TypeScript agent server that wraps tmux and speaks WebSockets. The backend is FastAPI on AWS with DynamoDB; it relays between the phone and your machine and never hosts an agent itself.

Mobile is a deliberately thin Capacitor shell that loads the deployed web app, so a fix reaches iOS, Android, macOS and the browser by deploying the web layer. The binary changes only when something native does.

macOS needed its own answer: Capacitor cannot build for it, because the Capacitor xcframework ships no Mac Catalyst slice. So the Mac app is a small AppKit window around a WKWebView, Developer ID signed, notarized and distributed directly.

Challenges we ran into

The first version was easy to build and impossible to give to anyone. All I wanted was for the agent CLIs on my own machine to expose their terminals, so that I could drive them from any browser or from my phone, and a private VPN with SSH does that in an afternoon. It worked. It also meant a wall of SSH config before anything ran at all, and it meant that anyone who wanted in had to be on my private network first. A colleague does not join your VPN to read one terminal. The architecture that made it simple for me was the same one that made it closed to everybody else.

So the VPN came out and AWS went in: a relay in the middle, every machine and every agent paired to it, and a phone that never needs a route to the machine itself. That fixed reach and collaboration in a single move, and it created the problem that took the longest to solve.

A terminal is not a request-and-response API. It is a keystroke and an echo, and a person feels the gap between them. Giving up SSH meant giving up a transport built for exactly that, and rebuilding it on WebSockets across a relay hop that was now ours to answer for. The first working version of that path took as much as 400 ms from keypress to character on screen: usable the way a bad phone line is usable, and enough to make the product feel like a remote control rather than a terminal. It runs at four to five milliseconds today. Getting there was the hardest engineering in the project and none of it was one clever trick. It is in how frames are batched, what is sent eagerly and what is not, how the connection is kept warm, and how much work is finished before a key is ever pressed.

The reason that work is slow is that it cannot cost you safety. SSH hands you authentication and an encrypted channel for free. A relay you built yourself hands you neither, and you have to put both back while the whole latency budget is single-digit milliseconds. That is the part that rewards knowing how the layers underneath actually behave, from the handshake to key handling to what each extra round trip really costs, and it is why I would still choose the relay over the VPN: the machines stay yours, the keys stay yours, and the result is something you can hand to a friend.

Three smaller problems each cost days, and every one of them was silent rather than loud. A Subscribe button that hung forever, because the payment plugin object is a thenable: return it from an async function and the runtime tries to resolve it as a promise that never settles. A sign-in that failed only for accounts outside the US, because Google's flow hops to a per-region host mid-redirect, that host was not in the WebView's allowlist, and Capacitor answers an unlisted host by handing it to Safari without a word. And a 62px dead band under the keyboard, where the obvious fix makes it worse, which we only knew because we measured all three modes on a real device.

Accomplishments that we're proud of

A real terminal on a phone. Not a chat log of a terminal: the tmux session, the scrollback, the keys, usable one-handed on a train.

Four platforms from one codebase, with a shell thin enough that shipping a fix to all of them is a web deploy.

A Mac app that is signed, notarized, stapled and accepted by Gatekeeper, distributed straight from our own site with no store in between.

Close to two thousand tests, and a habit behind them: every bug above is pinned by a test that fails if the fix is undone. The silent ones especially, because a failure that prints nothing is the one nobody notices twice.

And the loop closing. The app is now the way we build the app.

What we learned

Measure the device, not the documentation. Every bug above reads as impossible from the source and took ten minutes to see on a real phone.

We also learned when to stop. On-device neural text-to-speech with voice cloning (VoxCPM, 4-bit MLX) ran at 0.4x realtime on an iPhone and was killed by the OS at about 1.9 GB. We shelved it in a day and wrote down the numbers, rather than spending a week proving them again.

What's next for Avengents AI

Push notifications, so an agent can say it needs you instead of waiting to be found. An Apple Watch companion for the answer that is one word long. And the App Store release, which is in review now.

Built With

Share this project:

Updates

Submission history