doodleduo

• 

• 

Inspiration

AI agents are becoming increasingly capable of using websites, but most agent experiences still feel like work: fill out a form, search for something, click a button, complete a workflow.

We wanted to explore a different question:

What if a website was designed to play with an AI agent?

DoodleDuo started from the simplicity of drawing games like Skribbl: one player draws, the other guesses. We replaced the second human with an AI agent and used WebMCP to give that agent a native way to see the game, understand its state, draw on the canvas, make guesses, and chat with the human.

The result is a small experiment in human-agent play: two minds, one canvas.

What it does

DoodleDuo is a browser-based drawing game where you play directly with an AI agent.

During a standard game, the human and agent take turns being the artist and the guesser.

When you draw, the agent can:

• Read the current game state
• Request an image of the drawing canvas
• Interpret the drawing using a vision-capable model
• Submit guesses
• Chat with you during the game

When the agent draws, it receives the word and creates the drawing using structured vector strokes through WebMCP. You then try to guess what it is drawing.

The game includes:

• Easy, Normal, and Hard difficulty modes
• Timed rounds and scoring
• Alternating human/AI turns
• Custom word collections
• Custom 2–12 round games
• A cooperative "I draw every round" mode
• A local Doodlebot demo that works without an AI model
• A live conversation between the human and agent
• Tracking for WebMCP tool activity and canvas reads

The important part is that the agent is not just clicking around the interface using coordinates. DoodleDuo exposes the game itself as a set of structured capabilities that an agent can understand and invoke.

How we built it

At the center of DoodleDuo is a normal browser-based drawing game built around an HTML canvas and a stateful game loop.

We then added a WebMCP layer that exposes six tools:

• doodle_join — connects the agent to the game
• doodle_get_state — returns the current round, turn, timer, drawer, and scores
• doodle_get_canvas — gives the agent the drawing as a 1000 × 700 PNG along with optional stroke information
• doodle_draw — lets the agent draw validated vector paths onto the canvas
• doodle_guess — lets the agent submit a guess
• doodle_chat — lets the agent participate in the game conversation

Instead of asking a vision model to reason over an entire webpage screenshot, doodle_get_canvas exports only the drawing canvas. The UI, secret word, controls, chat, and other potentially revealing information remain outside the image sent to the agent.

For drawing, we took the opposite approach. Rather than having the agent simulate a mouse, the model sends structured strokes containing points, colors, and widths. The browser validates those strokes and renders them onto the canvas.

We also built a JavaScript RPC bridge alongside the WebMCP interfaces so different browser-agent environments can interact with the game.

For people who don't have a compatible agent connected, DoodleDuo includes Doodlebot, a completely local JavaScript demo player. It uses predefined procedural drawings and simple shape heuristics, so the core experience can be tried without spending model tokens.

The production version is deployed on Cloudflare, giving us a fast, globally accessible deployment with very little infrastructure required for the game itself.

Challenges we ran into

One of the hardest problems was surprisingly not drawing — it was fair play.

An AI browser agent can potentially see much more than a human player. If we simply gave the agent a screenshot of the webpage, it could accidentally read the secret word instead of interpreting the drawing.

We therefore had to think carefully about the boundary between information the agent needs and information it can technically access.

Our WebMCP tools intentionally omit the human's secret word and custom word list. The agent receives the canvas directly rather than a screenshot of the page, and it must request the canvas before making its first guess in a round.

Another challenge was coordinating a real-time game with an asynchronous agent. Drawing, guessing, timeouts, skips, round transitions, and human actions can all change the state while an agent is reasoning. Every tool therefore has to validate whether an action is legal for the current turn instead of assuming the agent always has fresh state.

We also had to handle two very different representations of a drawing:

Pixels for understanding, vectors for creation.

Vision models work naturally with an image of the canvas, while generating a drawing is much more reliable when the model can produce structured strokes rather than trying to control a mouse pixel-by-pixel.

Finding that separation made the interaction significantly cleaner.

Accomplishments that we're proud of

We're proud that DoodleDuo feels like an actual game rather than a WebMCP demonstration with a game wrapped around it.

The WebMCP integration remains mostly invisible to the human player. You draw, guess, laugh at bad drawings, and compete for points while the agent interacts with the same game through structured tools.

We're also proud that the project demonstrates several different agent capabilities inside one simple interaction:

vision → reasoning → tool use → structured generation → conversation

An agent can look at an image and guess what a human drew, then one round later receive a concept and generate its own drawing for the human to interpret.

We also built a zero-model-token local demo, custom games, multiple difficulty levels, scoring, turn validation, secret isolation, and a detailed transparency page explaining exactly what information the AI receives.

Most importantly, we built an experience where WebMCP isn't being used to automate a task.

It's being used to create a new kind of interaction between a human and an agent.

What we learned

The biggest thing we learned is that WebMCP is more interesting as an interaction layer than simply an automation layer.

Giving an agent a tool like doodle_draw is fundamentally different from telling it to "move the mouse around the canvas." The tool describes the semantic action the website supports while the website remains responsible for validating and executing it.

We also learned how important tool design becomes when the model is part of an interactive system.

Small decisions — what state to expose, when a secret should be omitted, whether an image or vectors should be returned, and what actions should be rejected — have a large impact on how reliably an agent can participate.

Another lesson was that WebMCP doesn't magically solve trust or security. A browser agent may still have access to screenshots, DOM inspection, or other browser capabilities. Applications need to explicitly think about what constitutes fair or safe agent behaviour instead of treating tool boundaries as a security sandbox.

Finally, building DoodleDuo made one idea very tangible:

Websites no longer have to be interfaces designed exclusively for humans.

They can expose a second interface — structured specifically for agents — while both humans and agents interact with the same underlying application.

What's next for DoodleDuo

DoodleDuo started as a human-vs-agent drawing game, but we think it could become a much broader playground for testing how humans and browser agents collaborate.

Next, we'd like to explore:

• Agent vs. agent matches, where two different models draw and guess against each other
• Multiplayer rooms, allowing humans and agents to form teams
• Agent tournaments, comparing how different vision models perform on the same drawings
• Replay mode, showing exactly which canvas observations, guesses, and tool calls happened during a round
• Richer drawing tools, including shapes, fills, layers, and more expressive brushes
• Shared WebMCP game primitives that could be reused to build other human-agent browser games
• Agent evaluation modes, turning DoodleDuo into a lightweight benchmark for vision, tool use, spatial reasoning, and multimodal interaction

Longer term, we'd like to explore the idea behind DoodleDuo beyond drawing games:

Built With

• chatgpt
• codex
• javascript
• react