muselet

muselet architecture
changing video speed on premiere pro
adjusting wifi and brightness at the same time
buying batteries for me

Inspiration

Computer-use agents today stream your screen to a cloud model and take seconds per click. That's costly and a huge privacy problem: your whole screen leaves the machine. I wanted the opposite, an agent that lives on your Mac, reacts like a keyboard shortcut, and keeps every pixel local.

What it does

Hold right option, say what you want, and muselet does it in your apps: it opens apps, clicks, types and uses keyboard shortcuts, then checks that the outcome actually happened. Every model runs on Apple Silicon, with no cloud inference.

It's built for anyone who'd rather speak than click: power users chaining tasks, and people for whom a mouse and keyboard are hard to use.

How I built it

A System 1 / System 2 design: a fast trained router for instant commands, and a reasoning planner for everything else.

Voice: Parakeet TDT v2 (FluidAudio, Core ML) on the Apple Neural Engine, with streaming partials.
Router (fine-tuned by me): Qwen3.5-4B + my LoRA, 4-bit MLX. One forward pass scores 43 options (native OS skills plus MULTI_STEP / ASK / REFUSE / ANSWER).
Training data: synthetic utterances with ASR-style noise, soft labels from a Qwen 35B-A3B teacher, LLM-judge audit and adjudication, and a decontamination gate against every eval set. Trained on Tinker.
Planner: local Qwen3.5-4B, 4-bit with thinking enabled, chooses one UI tool at a time from fresh accessibility and on-device OCR observations, then checks the resulting state.
Hands (Swift): model-selected AX actions, keys, clicks and AppleScript/shell, through the Accessibility API. A ghost cursor flies to each target first, so you see intent before action.
Vision fallback: eyes-4b with my two-stage zoom (≤1 MP), used only when the accessibility tree is blind.
Verify: checks the outcome against fresh screen state and replans if it isn't met. DONE is never trusted.
Safety: send, delete, buy, post and shell actions need spoken confirmation; REFUSE is a first-class action.
Plumbing: a Swift menu-bar app and a Python brain talk over a localhost WebSocket and share one skill catalog. Replies are ≤12-word text bubbles: glanceable, no TTS lag.

Challenges I ran into

Instant vs. smart. Single commands must feel instant; multi-step tasks need reasoning. Splitting the router from the planner gave me both without compromising either.
Offline scores vs. live performance. I also fine-tuned a planner LoRA, but offline scores do not establish which configuration works best on a real desktop. The demo currently uses the base 4B with thinking enabled; a controlled live comparison is still next.
Agents touching real machines. An early test typed into a real document. Now live cases run only through a runner that enforces each case's preconditions, and risky actions always ask first.

Accomplishments that I'm proud of

A custom-trained router that decides in ~60 ms (median) with 94.5% accuracy on route_v2, fully on-device.
Fine-tuned for under $150 of Tinker compute, end to end.
My two-stage zoom lifts eyes-4b grounding from 36.0% to 63.5% on a 200-sample ScreenSpot-Pro subset at a 1 MP image cap.
Zero cloud inference: speech recognition, routing, planning and vision all run locally on Apple Silicon.
An agent you can trust to touch your Mac: visible intent (ghost cursor), confirmation gates, and outcome verification on every task.

What I learned

Small local models are enough when the system around them carries the load: a tight action space, accessibility-first observation, code (not the model) for parsing, and verification on every step.

Hard part is not generating a good next click. It is knowing whether that click changed the right thing and recovering when the app does not expose enough information.

What's next for muselet

Run controlled live comparisons of the base and fine-tuned planners, improve reliability in canvas-heavy apps and turn more observed failures into permanent regression tests.

I also want to expand the instant-command catalog while keeping complex app tasks AI-planned through generic tools.