💡 Inspiration

Physical computing and IoT have always promised an exciting world of creativity—turning code into blinking LEDs, motor movements, robotics, and environmental sensors. Yet for beginners, educators, and even seasoned makers, the onboarding experience is plagued by toolchain friction:

• Hunting down and installing OS-specific USB-to-UART drivers (CH340, CP2102, FTDI).
• Configuring desktop IDEs like Thonny, Arduino IDE, or heavy VS Code toolchains.
• Juggling serial baud rates, COM port drops, and manual file transfer scripts.

With modern browser capabilities like the Web Serial API and File System Access API, we realized that the entire embedded development lifecycle could run 100% inside the browser with zero installation.

When WebMCP was introduced as an open standard for agentic web applications, we asked a fundamental question:
Why should AI agents only interact with web forms, spreadsheets, and SaaS tools? What if an AI agent had physical hands on hardware?

Cobweb was born to bridge this divide—creating a zero-install MicroPython IDE where humans and AI agents collaborate seamlessly to probe, code, execute, and debug physical microcontrollers.

---

⚡ What it Does

Cobweb turns any Chromium browser into a full-featured MicroPython development environment that connects directly to microcontrollers (such as Raspberry Pi Pico, ESP32, or BBC micro:bit) over Web Serial at $115{,}200\text{ baud}$.

Key capabilities include:

• Zero-Install Python IDE: Code editor powered by CodeMirror 6 with syntax highlighting, automatic indenting, and multi-file local storage.
• Interactive xterm.js REPL: Direct serial connection with live terminal streaming, ANSI color rendering, and raw-paste mode.
• Device Filesystem Management: Browse, read, write, edit, and delete files stored on the microcontroller's onboard flash memory.
• Deep WebMCP Integration: Exposes 15+ fine-grained, structured tools to external AI agents via document.modelContext.registerTool.
• Human-in-the-Loop Safety: Every mutating agent action (flashing files, modifying code, running hardware commands) triggers an interactive approval modal with live side-by-side diff previews before anything touches the physical board.
• Board Hardware Notes: Persistent scratchpad where the agent and user track pin mappings (e.g., $I^2C$ SDA/SCL pins, SPI buses, ADC channels, and sensor addresses).

---

🛠️ How We Built It

Cobweb is built as a pure client-side Progressive Web App (PWA) with no backend servers or cloud dependencies:

┌─────────────────────────────────────────────────────────────┐
│                    External AI Agent                        │
│             (ChatGPT / Chrome with WebMCP)                  │
└──────────────────────────────┬──────────────────────────────┘
                               │ WebMCP JSON-RPC / Tools API
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                       Cobweb WebApp                         │
│  ┌───────────────────────┐       ┌────────────────────────┐ │
│  │   WebMcpBridge.tsx    │◄─────►│    Tool Registry       │ │
│  │ (use-webmcp-tool)     │       │ (15+ Structured Tools) │ │
│  └───────────┬───────────┘       └───────────┬────────────┘ │
│              │                               │              │
│              ▼                               ▼              │
│   ┌─────────────────────┐       ┌────────────────────────┐  │
│   │ CobwebApproval Modal│       │   ReplInterface (TS)   │  │
│   │   (Live Diffs)      │       │ (Raw REPL State Mach.) │  │
│   └─────────────────────┘       └────────────┬───────────┘  │
└──────────────────────────────────────────────┼──────────────┘
                                               │ Web Serial API
                                               ▼
                         ┌───────────────────────────┐
                         │   Physical Board (Pico)   │
                         └───────────────────────────┘

1. The WebMCP Bridge (WebMcpBridge.tsx)

We utilized use-webmcp-tool to dynamically expose our tool registry to the browser's model context:

useWebMCP({
  name: def.name,
  description: def.description,
  inputSchema: def.parameters,
  async execute(args) {
    if (def.requiresApproval) {
      const approved = await requestApproval({ id, name: def.name, args });
      if (!approved) throw new Error('Tool execution rejected by user.');
    }
    return await tool.call(args, {});
  }
});

2. High-Performance Serial Communication (ReplInterface.ts)

MicroPython communication relies on a custom finite state machine implementing the raw-REPL protocol:

1. Raw Mode Entry: The host sends \x01 ($\text{Ctrl-A}$) to enter raw REPL.
2. Raw-Paste Windowing: To prevent buffer overrun on microcontrollers with constrained RAM ($\approx 264\text{ KB}$ on RP2040), we implement dynamic flow control: $$\text{Chunk Size} \le W_{\text{window}}, \quad \text{replenished by } \text{\textbackslash x01 bytes}$$
3. Execution & Stream Separation: Code is executed via \x04 ($\text{Ctrl-D}$), returning structured stdout and stderr delimited by end-of-transmission bytes: $$\text{Banner} \longrightarrow \text{OK} \longrightarrow \text{stdout} \longrightarrow \text{\textbackslash x04} \longrightarrow \text{stderr} \longrightarrow \text{\textbackslash x04}$$
4. Queue Serialisation: An AsyncBlockingQueue<Uint8Array> safely decouples serial reads from async execution promises while mirroring bytes to xterm.js.

---

🧗 Challenges We Ran Into

1. Single-Channel Serial Contention

The Web Serial port is a single, non-multiplexed stream. If a user is typing in the REPL while an AI agent attempts to run get_board_info or read_device_file, raw bytes interleave and corrupt the state.
Solution: We built a single-slot promise lock chain for all read/write/eval calls (send, sendRaw, reset), ensuring strict atomicity without blocking UI rendering or serial event dispatching.

---

🏆 Accomplishments That We're Proud Of

• 100% Client-Side Architecture: Zero backend servers, zero telemetry, zero credential relay. Everything runs directly between the user's browser and their USB port.
• True Agentic Physical Computing: An AI agent running in ChatGPT or Chrome can inspect the board identity (os.uname().machine), write sensor drivers, execute test snippets, and diagnose hardware errors in real time.
• Rich Human-Agent Co-Pilot Experience: Instead of replacing the human, WebMCP in Cobweb creates a fluid dialogue where the human handles breadboarding and hardware wiring, while the agent handles MicroPython syntax, pin registers, and driver debugging.

---

📚 What We Learned

• WebMCP is a Game Changer for Browser Capabilities: Traditional desktop agent tools require complex native MCP servers (stdio/SSE over localhost). WebMCP brings tool calling directly to the browser DOM and Web APIs, making cutting-edge capabilities like Web Serial, Web Bluetooth, and File System Access instantly accessible to AI agents.
• Structured Tools Beat Vision/DOM Scraping: Prior to WebMCP, agents navigating web IDEs had to guess button positions or scrape xterm canvases. With WebMCP, tool calls are typed, deterministic, and instant.

Built With

• micropython
• react
• vite
• webmcp
• webserial