Whiteboard Agent

Inspiration

Most AI diagram tools work in batches. You give a prompt, wait, and get a static result.

I wanted something more interactive: a whiteboard where a person and an AI agent can work together in real time. WebMCP made this possible by allowing the website to expose tools through document.modelContext, which an AI agent can discover and use without a plugin or API key.

What it does

Whiteboard Live is a live whiteboard with 32 WebMCP tools. An AI agent can discover these tools and build diagrams directly on the canvas while you watch.

It supports:

• Text, shapes, sticky notes, illustrations, icons, equations, code blocks, sketches, and connectors
• Live step-by-step drawing instead of batch rendering
• Human and AI working on the same canvas
• Light, Dark, Sepia, and Blackboard themes
• Voice input and typed prompts
• Up to 20 levels of undo

How I built it

The app uses React + SVG. SVG is used for connectors and strokes, while regular DOM elements handle draggable and resizable content.

WebMCP tools are registered through document.modelContext.registerTool(). When an agent calls a tool, the canvas updates immediately with a reveal animation.

For visuals, I use a four-level fallback system:

1. Local SVG illustrations
2. Iconify for icons
3. AI-generated SVGs using Minimax for unknown concepts
4. Emoji as a final fallback

Generated assets are cached so they can be reused.

The apply_visual_scene tool also has an automatic layout engine. The agent only decides what should be shown, while the layout engine handles the positions. It supports layouts such as horizontal processes, radial diagrams, hierarchies, comparisons, timelines, two-column layouts, and architectures.

The app also includes a WebMCP polyfill, so it can work in regular browsers through the built-in prompt input.

Challenges

The main challenges were making batch operations safe, finding visuals quickly, supporting browsers without WebMCP, and keeping the AI from having to calculate pixel coordinates.

For batch operations, the canvas state is saved before changes and automatically restored if something fails.

For visuals, the fallback system makes sure every concept gets something useful without blocking the agent.

The polyfill allows the same experience in browsers that do not support WebMCP.

The semantic layout system handles positioning so the agent can focus on the content instead of coordinates.

What I learned

WebMCP creates a different way for humans and AI to work together. The AI can discover website tools and interact directly with the page instead of just returning an answer.

The biggest insight for me was the shared canvas. The AI can build something, the human can edit it, and the AI can continue from the updated canvas.

It feels less like using an AI tool and more like working with an AI teammate.

Built With

• katex
• openrouter
• radix-ui
• react
• rough-js
• tailwind-css
• tanstack-start
• typescript
• vercel
• vite
• web-speech-api
• webmcp