PixelForge

An agent-native pixel art studio where humans and AI agents operate the same editor through WebMCP — 19 tools, one shared canvas, every action visible and undoable.

Comment

Inspiration

AI "editors" today are one-shot generators: you prompt, you get a file, and you fix it by hand. The agent never touches the actual software. We asked the opposite question — what if the agent could operate a real creative tool, the same way a human does?

A pixel editor turned out to be the perfect answer. Pixel art is 100% structured state — a grid of colors, layers, frames, a palette — which maps exactly onto tool schemas. No vision models, no guesswork: an agent can read the canvas as exact pixel data and edit it through the same primitives a human uses.

What it does

PixelForge is a complete, Aseprite-inspired pixel art studio: pencil / eraser / flood fill / color picker, brush shapes (circle, square, line) from 1–64px with pixel-perfect strokes, layers (duplicate, merge down, flatten), a cel-based animation timeline with playback, onion skinning, palette sorting & gradients, canvas resize, and PNG / animated GIF / sprite-sheet export.

The same studio is also an agent workstation. ChatGPT — or any WebMCP client — can inspect the canvas, batch-draw hundreds of pixels, manage layers and frames, recolor entire animations, and export finished assets, through 19 structured WebMCP tools registered via document.modelContext.registerTool.

Try it: open the live URL in ChatGPT's in-app browser and ask:

"Create a cute green slime with a simple face, then turn it into a 4-frame idle animation."

How we built it

React + TypeScript + Vite + Zustand + Tailwind — fully client-side, no backend.

The core rule: one action layer, two users. All 19 WebMCP tools call the exact same store actions as the mouse and keyboard. There is no separate "AI drawing" code path — so every agent edit is history-tracked, replayable, and undoable by the human.

  • Tools use strict JSON schemas with batched operations (draw_pixels takes an array of pixels — no one-call-per-pixel)
  • Layer/frame references resolve by id or name, so agents never need to track internal ids
  • Every tool returns structured JSON: {success, operation, detail} or {success:false, error:{code,message}} — so the agent can reason about failures instead of guessing
  • Read-only tools (get_editor_state, get_region) let the agent see the canvas as exact pixel data before acting
  • A live Agent Activity panel shows every tool call with HUMAN/AGENT badges — and every mutation, human or agent, is undoable

38 unit tests cover the editor actions and the tool contracts (schema validity, batched draws, structured errors, undoability of agent mutations).

Challenges we ran into

  • Making agent tool calls render live on the canvas while still committing as one undoable step
  • Keeping agents effective across animation frames: layer references are remapped by name, so "draw on the Eyes layer" keeps working after frame switches
  • Designing batched pixel operations that survive strict JSON-schema validation
  • Testing across environments (ChatGPT in-app browser, Chrome with the testing flag) — plus building an honest debug bridge for browsers without WebMCP, clearly labeled so nothing is faked

Accomplishments that we're proud of

  • The full creative pipeline runs on WebMCP. A judge can ask ChatGPT: "Create a green slime → make the eyes smaller → turn it into a 4-frame idle animation → make the palette warmer → export a sprite sheet" — and watch every step execute as real tool calls on the live canvas, including 411 pixels placed in a single batched call.

  • A complete product, not a tech demo. In 9 days we built an entire Aseprite-style studio from scratch: cel-based animation timeline, onion skinning, brush shapes with pixel-perfect strokes, palette tools, three export formats — all sharing one action layer with the agent.

  • Trust by design. Every agent operation appears in the activity feed with an AGENT badge and sits in the same undo history as human edits. The human can take back anything the agent did — visibility and control are built in, not bolted on.

  • Verified, not claimed. 38 unit tests cover the editor actions and tool contracts, and the 19 tools were confirmed live in both target environments: ChatGPT's in-app browser and Chrome with the WebMCP testing flag.

What we learned

Strict schemas beat clever prompts. Because every tool has precise input validation and returns structured errors (INVALID_COORDINATE, LAYER_NOT_FOUND, NO_SELECTION), the agent self-corrects instead of hallucinating. And sharing one action layer between human and agent turned out to be the cheapest possible way to earn trust: visibility + undoability come for free.

What's next for PixelForge

Declarative WebMCP tool registration, agent-driven tileset workflows, multi-sprite projects — and WebMCP support for tiled drawing mode.

Built With

Share this project:

Updates

Submission history