Shabang

Inspiration

We were on Twitter the day before the hackathon when Jev launched. The idea had been sitting on the tip of our minds already: make computer use feel like autocomplete, where the right next action appears instantly and you stay in control. What we did not have was a model that could make that call fast enough, and honestly enough, to gate on.

Most agents work remotely, reason over screenshots, or take several seconds per click. Jev returns typed decisions with calibrated confidence in a couple of hundred milliseconds, which is the difference between an agent that feels slow and one that feels like Cursor. It made the version of this idea we actually wanted suddenly buildable.

What it does

Shabang is a native macOS agent. It reads the accessibility tree of whatever app is frontmost, a web page, Spotify, Messages, Finder, and proposes the single thing you are most likely to do next. A purple ring marks it, a ghost cursor points at it, and right Command takes it.

It replies in Messages, plays and fullscreens video, opens the file you were reaching for, and fills job applications. On a form it plans every field in one batched decision, fills them as you advance, verifies each write, and stops on Submit without pressing it.

The interesting half is what happens when you say no. Every outcome is recorded, taken or refused. Locally it learns you immediately. To Sentry it goes as a value free envelope, because a suggestion you turn down is a labelled training example, and it is the only thing the product learns from.

A wrong suggestion is worse than none, so nothing appears unless confidence clears the gate.

How we built it

A native macOS menu bar app over the Accessibility APIs, with the shared prediction logic in TypeScript running through JavaScriptCore, and a loopback server holding the API keys.

Jev (TypeSafe) is the decision layer. Given normalized accessibility state and a filtered set of safe candidates, it picks the action and returns calibrated confidence. One batched call per form, never one per field, so the walk afterwards reads precomputed answers at zero cost per keystroke. On our 12 field application benchmark it was 100% correct at 266 ms median, across six distinct confidence values.

Baseten (GLM-5.3-Flash) writes every word the product produces, because Jev returns typed decisions and cannot generate text. iMessage replies come back in about 750 ms. Baseten also works as a decision provider, and since its API accepts logprobs but returns none, we built the confidence signal ourselves: the same schema constrained request fired four times in parallel, the first three valid answers voted, stragglers aborted. Agreement becomes the confidence. It scored 100% on both benchmark forms with zero wrong answers above the gate.

OpenAI vision handles controls that publish no accessible name. Honestly, it rarely fires: on every app we measured, almost nothing was nameless. The controls we assumed needed eyes turned out to be named, and our own classifier vocabulary was the real gap.

Sentry is the feedback loop, covered in its own section below.

An MCP war room. Midway through, our agents were working across machines and branches faster than we could coordinate by hand, and the problem is worse on an idea this tightly coupled: there are not many independent pieces to split, so two agents almost always want the same files. We built a small MCP service on Replit where each agent announces what it is touching and reads what the others are on. It is not part of the product. It is how the product got built without us constantly overwriting each other.

Challenges we ran into

Jev is text only and cannot be trained. Both facts shaped the build. Not multimodal means a screenshot is useless to it, so every decision had to come from accessibility state we normalized ourselves, and the one job that genuinely needs eyes went to OpenAI vision instead. Not trainable means there is no fine tune to fall back on: the only lever you have is the question you write. That turned out to matter more than we expected, and it is exactly where our biggest accuracy win came from.

Real apps fight you. Every one of these was found by measuring rather than guessing.

A press that lies. In any Chromium hosted window, Electron, CEF or a browser, a control publishes AXPress, returns success, and does nothing at all. Our accept reported ok and the app never moved, which is the worst possible failure: the suggestion was right and the confirmation was wrong. We now click for real there, and a list row gets two clicks, because one only selects it.
A file input that hides. Chromium publishes <input type=file> as a plain button with no upload subrole. Three separate checks wanted that subrole, so resume uploads silently did not exist in Chrome while working perfectly in Safari.
A field with no name. On a real application the Location field published no title, no aria-label and no aria-labelledby. Its label sat in the text before it, and the required asterisk, rendered as a CSS pseudo element, arrived as its own node and displaced the label. Every other field filled. That one always failed.
A player with no media element. Chromium publishes no video node, so a video page classified as a feed and the best thing we could offer after opening a video was an advert in the sidebar. We now recognise a player by its transport bar.

Pivots. We started as a Chrome extension and abandoned it, because an extension can never see Messages or Spotify and the whole premise is that this works anywhere. We killed an earlier "do it twice and it does the rest" batch automation in favour of single next action prediction, which is both safer and the thing people actually wanted. And we changed the accept key from Tab to right Command after discovering that Tab is the most overloaded key on the keyboard, and that right Option, our first replacement, collides with macOS Mouse Keys and with another app's hotbar.

How Sentry changed the code

Not monitoring bolted on. Sentry found four things we would not have found by reading the code.

We were optimising the wrong thing. We assumed the model was the slow part. Per span timing said our local ranker takes 0.06 ms, about 0.4% of one frame, while reading the accessibility tree takes roughly 160 ms. Capture is about 2,600 times slower than the thing we were about to optimise. The fix is incremental capture, not a faster model, and nobody guesses that.

The rejection stream was invisible. A walk where the user refused every suggestion logged as "0 accepted, 0 corrected", so the only signal the product learns from read as silence. Fixing it exposed a second bug hiding underneath: ghost.rejected was missing from our scrub allowlist, so the counts shipped as [redacted:key:0] while every test that did not assert on the value still passed.

We were blaming the model for our own prompt. Our committed benchmark reported Jev at 55% on ambiguous fields and we nearly wrote it up as a limitation. Reading which answers lost showed the question was wrong, not the model: we asked for "phone number" without saying whose, so on "Emergency contact phone" its answer was correct for the question we actually asked. Stating ownership took it to 90% and took wrong suggestions above the gate from four per call to zero. Since Jev cannot be fine tuned, this is the whole optimisation surface, and it turned out to be enough.

Worse, we had rigged our own comparison by accident: one provider's system prompt named two rows of the benchmark form, so it was handed the grader's rubric while Jev, which has no system prompt channel, got nothing. Removed, and the scores held.

Error monitoring caught three zombie servers. An EADDRINUSE issue turned out to be three server processes running, the oldest holding the port, which is why a code change appeared to have no effect. We hit the same class of bug again later: a server from a different working copy answering on the same port after we renamed a route, so every text request returned 404 and the drafts quietly stopped. The fix that stuck was making the outcome post report whether Sentry accepted it, so the loop can never be silently broken again.

Accomplishments that we're proud of

A native agent that is fast and cautious at the same time.

It fills a real application end to end: 14 fields, both long form answers drafted, a resume attached through the macOS open panel in 2.1 seconds, and Submit left untouched.

We treat confidence as a product feature rather than decoration, and we can show why. In the same harness, on the same forms, gpt-4o-mini returned exactly one distinct confidence value across every call. A constant is not a signal, so 96 wrong answers passed a 0.7 gate untouched. Jev gives six distinct values on one form and nine on another, and is 0.98 sure when it is right against 0.53 when it is wrong. That gap is what the gate turns into "show this, hold that back".

Six Sentry products in use, five of them beyond error monitoring, with a schema built from closed string literal unions so a label, URL or typed value cannot be represented at all, not merely scrubbed.

What we learned

Raw accuracy is not enough. Calibration matters more, because an agent has to know when to stay quiet.

The model should not do everything. Jev chooses among safe semantic actions. Native code observes the UI, performs the action, and verifies the result. That separation is why a press that lies gets caught instead of silently reporting success.

When you cannot train a model, the question is the product. We spent more time on how we asked than on what we asked, and it paid better than any model change would have.

Telemetry is product feedback, not debugging data. Looking at accepted, dismissed, corrected and rejected outcomes found broken assumptions, fixed prompts, and told us which bottleneck was real.

What's next for Shabang

Incremental capture through AXObserver, so a focus change costs one node instead of four hundred. Wider workflow coverage, stronger role based memory across similar interfaces, better text generation and vision fallback, and continued work on the safety model around high impact actions.

The goal stays the same: the fastest useful next action, at the right moment, with the user firmly in control.