TinyPilot WebMCP

Inspiration

Remote support often becomes a conversation between someone who can see a problem and an expert explaining which keys to press. TinyPilot already brings a remote computer's display, keyboard, and mouse into a browser. We extended that shared console so an AI agent can take precise, inspectable actions while the person watches and can intervene at any time.

What it does

TinyPilot WebMCP exposes the dashboard's major operations as 75 browser-native tools. They cover console status and screenshots, keyboard and mouse input, display and video controls, diagnostics, networking, security, updates, power, and installed TinyPilot Pro capabilities. The tools register when the normal dashboard loads and are discovered by the WebMCP host.

The demo video proves the integration against a physical TinyPilot Pro appliance and connected computer. Given a goal such as navigating Wikipedia from Common sunflower to Family Matters, the agent observes the same display as the person, sends its navigation through TinyPilot's WebMCP controls, and reaches the requested page on the attached computer.

About the live demo

The hosted demo uses an isolated simulated desktop so every judge can safely test the WebMCP tools without sharing access to one physical machine. In real use, the same integration runs on a TinyPilot device connected to the target computer through video and USB. The demo video shows the integration operating through physical TinyPilot hardware.

Why WebMCP matters

TinyPilot connects to the target's physical video output and USB input, so the target only needs to accept a monitor, keyboard, and mouse. It does not need a custom agent, browser extension, or remote-control service. This makes agentic computer use possible for operating-system installers, recovery consoles, locked-down devices, legacy systems, and remote servers where target-side software is difficult or impossible to install.

The relevant context already exists in the TinyPilot page: the target computer's current display, the operator's authenticated session, and the controls the person is using. WebMCP exposes those capabilities at the moment they are useful. The person and agent share one dashboard and see the same outcome.

How we built it

The top-level dashboard registers tools with document.modelContext.registerTool. Each tool has a JSON schema and runtime validation. Execution reuses TinyPilot's existing authenticated frontend actions and backend interfaces, supports cancellation where applicable, and returns structured results. Sensitive values are write-only and omitted from adapter logs. Optional Pro adapters discover installed controllers at runtime without redistributing TinyPilot's licensed implementation.

The repository includes the implementation, 84 JavaScript tests, the full capability map, a restricted-container demo, deployment and rollback tooling, and the upstream MIT license. The physical install was verified by exact source hashes, active service checks, native browser discovery, and a safe live status call.

Challenges and what we learned

The hardest part was preserving TinyPilot's existing security and interaction model while exposing a broad control surface. We mapped each dashboard capability to a typed tool, validated inputs again at execution time, made optional Pro features discoverable only when installed, and kept secrets out of results and logs. We also learned that WebMCP fits remote KVM especially well: the browser already contains the correct authenticated context and the human remains in the loop through the same live console.

What is new

TinyPilot Community is the open-source foundation, with its existing authorship and license retained. The challenge branch begins at upstream commit 5f34d8a66d05fb8ca3625e7458915e55cfb7fac7 and contains the WebMCP integration plus the public demo, tests, documentation, and deployment tooling.

Built With

• cloudflare
• docker
• javascript
• tinypilot
• webmcp