AgentDraw

Inspiration

Most agent integrations live behind a chat box or a server API. I wanted a shared visual workspace where a person can sketch, select, move, and annotate freely while an AI agent can understand the same canvas and make precise, reversible changes. AgentDraw turns Excalidraw into that collaborative surface with WebMCP.

What it does

AgentDraw is a local-first whiteboard that exposes eight site tools to an agent: capabilities, structured canvas inspection, atomic patches, selective patch rollback, focus, visual capture, export, and scene import. The agent can create labelled shapes, bound arrows, frames, images, and live HTML prototypes. A person can draw freehand marks around a component; inspection reports nearby element IDs so the agent can understand references such as “this step.” Human and agent edits share Excalidraw’s undo history, while revision and epoch checks reject stale writes instead of overwriting recent human work.

A particularly useful workflow is app prototyping: ask the agent to draw a flow, circle a step by hand and request a correction, then ask for an interactive form beside the diagram. The prototype runs as sandboxed HTML inside the canvas. The agent can later replace its source, export the result as SVG or an Excalidraw document, or revert only one of its own patches without removing the person’s work.

Challenges

The hardest parts were preserving Excalidraw semantics while allowing structured agent edits, coalescing noisy pointer events into useful revisions, keeping arrows and labels attached during moves and resizes, and preventing selective rollback from clobbering later human changes. Another challenge was giving the agent both structural understanding and visual grounding without dumping an enormous canvas into context.

What I learned

WebMCP is especially powerful when the browser page itself is the shared state. It removes a separate MCP server and lets the agent work in the same authenticated, visual session as the person. I also learned that good agent tools need explicit concurrency, compact read models, strong validation, and reversible writes—not just a large set of actions.

Built With

• excalidraw
• indexeddb
• react
• typescript
• vite
• webmcp