RemoteCtrl

Give access to the task, not the machine.

Comment

RemoteCtrl

Access to the task. Not the machine.

Inspiration

We kept running into the same wall while building AI browser agents: every tool out there makes you pick your poison.

RustDesk and AnyDesk? Hand over the entire machine just to get help with one browser task. Cloud browser agents like Browserbase or browser-use? You get a clean, empty browser that has no idea who you are — so you're re-logging into everything from scratch. n8n and classic RPA? Replays the exact same clicks forever, and snaps the second a button moves. And most "AI safety" right now is just a system prompt begging the model to behave, with zero enforcement if it decides to wander off.

None of these tools are wrong, exactly — they're just optimizing for the wrong unit of access. You shouldn't have to hand over your whole machine, or your whole session, just to delegate one task to an agent (or a support person).

So that's what we built: access to one approved browser task, not the machine. You keep your real, already-logged-in browser. The agent gets scoped, runtime-enforced access to exactly the work you approved — nothing more. You can step in and take over at any second, and once a run succeeds, it becomes a reusable workflow.

What it does

RemoteCtrl is a task-scoped browser coworker — Electron, React, TypeScript, Playwright, WebRTC — that puts a live browser next to an AI chat console, with real enforcement running underneath.

It actually can't wander off. This is the part we're most proud of. Most agent "guardrails" are just prompt instructions — "stay on this domain" — which the model can ignore if it gets confused. We built a real runtime Policy Gate that sits inside the CDP/Playwright layer and checks every single action (navigate, click, type, keypress) against an allowed-domain whitelist and an action budget before it's allowed to touch the browser. Out of scope? Blocked before it happens, not after. (Right now this locks down AI actions and gives full human takeover — remote human input over WebRTC isn't scope-enforced yet, that's next.)

It doesn't choke on messy pages. Dumping raw HTML into an LLM context is slow and expensive, so we snapshot the page, tag clickable stuff with clean numbered labels, and hand the model a lightweight markdown summary instead. When it says "click 3," we resolve that to a selector that'll actually survive a re-render — prioritizing stable IDs and semantic attributes over anything brittle.

It knows when to stop and ask. CAPTCHA, 2FA, some high-stakes confirmation — the agent doesn't try to be clever about it. It pauses, throws an interactive prompt into the chat, and picks up exactly where it left off once you've handled it.

It turns good runs into workflows that heal themselves. Every successful agent session can be saved as a deterministic, replayable workflow. And when a site's layout changes and a saved step breaks? It doesn't just fail — it pauses, spins up a small recovery agent with the step's original description, finds the new element, finishes the job, and saves the repaired selector for next time.

Local-first, actually. API keys are encrypted at rest via your OS's native secure store. Voice commands run through a local Whisper model — fully offline, nothing sent anywhere.

How we built it

React (renderer)  →  Preload bridge (Zod-validated IPC)  →  Electron main process
                                                               ├─ Playwright/CDP  → your real browser
                                                               ├─ Policy Gate      → checks every agent action
                                                               ├─ Vercel AI SDK    → multi-provider LLM loop
                                                               ├─ Local Whisper    → offline voice input
                                                               └─ WebRTC/Socket.IO → remote operator streaming
  • Frontend: React 19 + Zustand + Radix/Tailwind — live browser panel, agent step feed, drag-and-drop workflow editor.
  • Security boundary: nodeIntegration: false, contextIsolation: true, every IPC call schema-validated before it's allowed to execute. No shortcuts here.
  • Browser engine: Playwright over CDP, attaching directly to real local Chrome profiles — so cookies and sessions are already there, no re-auth needed.
  • Remote sessions: host streams over WebRTC via a lightweight Socket.IO signaling server; a Chrome extension bridges other tabs into the workflow recorder over a local WebSocket.

Challenges we ran into

  1. Building a policy gate that's actually real. Telling a model "don't leave github.com" is easy. Building something that intercepts every tool call, checks domain scope, tracks a TTL and action budget, and can pause for async human approval — without race conditions between the AI loop and the browser thread — took real work.
  2. Numbered elements go stale instantly. The moment a page navigates or re-renders, those tidy [1], [2] indices mean nothing. We rebuild the snapshot after every state change and generate selectors that prioritize durability over convenience.
  3. Handing off from deterministic replay to AI recovery without infinite-looping. Getting a broken step to gracefully call in an AI sub-agent, fix itself, and persist the fix back to the saved workflow — cleanly, with bounded retries — took a few passes to get right.
  4. Making fast automation feel trustworthy to watch. Watching an agent click around your real, logged-in browser at speed is unsettling if you can't tell what it's about to do. We added an animated cursor overlay with a click ripple before every action, just so a human watching the screen doesn't lose the thread.

Accomplishments that we're proud of

  • Proving the core idea holds up: bounded, runtime-enforced AI task execution inside a user's real authenticated session is a genuinely different — and safer — shape of delegation than what exists today.
  • Watching self-healing actually heal: a workflow hits a broken selector, pauses, fixes itself, finishes the step, and remembers the fix. That's the whole thesis paying off.
  • Enterprise-grade bones for a hackathon build: strict Electron isolation, 100% Zod-validated IPC, OS-level credential encryption.
  • Fully offline speech-to-text with a verified local model and zero cloud audio transmission.

What we learned

  • Prompt policy ≠ runtime policy. Telling an LLM its boundaries helps with steering, but only runtime enforcement earns real trust.
  • Agents and workflows aren't competitors, they're a pipeline. Agents are great at figuring out a task once; they're too slow and expensive to run from scratch every time. Converting a successful run into a deterministic workflow — backed by AI healing when things break — is the right architecture.
  • Human-in-the-loop is a feature, not a fallback. Nobody wants an agent that guesses its way through 2FA. Giving it an explicit "ask" checkpoint builds real confidence fast.

What's next for RemoteCtrl

  • Full scope enforcement for remote humans — routing WebRTC mouse/keyboard input through the same policy gate that already governs the AI agent.
  • A more powerful workflow editor — branching conditionals, parallel steps, nested sub-workflows.
  • Real cross-platform distribution — signed, notarized builds for macOS, Windows, and Linux.
  • Enterprise-hardened signaling — TLS end-to-end, OAuth/SAML identity, audit trails, and team collaboration profiles.

Built With

Share this project:

Updates