CommandCanvas

A shared spatial workspace where people and agents turn rough thinking into structured, attributable output on the same live canvas.

Comment

Inspiration

Meetings produce sketches, decisions, tasks, and commitments, but the useful deliverable is often reconstructed afterward in another application. CommandCanvas lets the meeting itself produce structured output.

People, hands, collaborators, voice, and ChatGPT work against the same live spatial objects. Instead of sending an agent a stale transcript, CommandCanvas gives it bounded capabilities over the canvas everyone is using.

What it does

CommandCanvas is an infinite collaboration canvas built from semantic objects rather than untyped pixels. A participant can create notes, boards, schedules, sketches, diagrams, charts, tables, decisions, actions, and meeting packets; arrange them spatially; and see who changed what.

The central workflow starts with rough human thinking. A participant draws a multi-stroke sketch while explaining it, then asks the agent to make it usable. CommandCanvas preserves the original, creates a separate structured result beside it, records provenance, and adds an immutable receipt. The result is not limited to architecture: the same system supports flowcharts, generic diagrams, bar, line, and pie charts, tables, boards, and written summaries.

Every input follows one path:

Hands / voice / pointer / collaborator / WebMCP
                         ↓
                validated capability
                         ↓
               canonical mutation
                         ↓
          revision + receipt + Supabase sync

This lets a hand gesture, a remote collaborator, and ChatGPT change the same durable workspace without creating separate sources of truth. Mutations are visible and attributable. Destructive actions are recoverable. An agent may prepare a meeting packet and stage a send request, but the human host must approve the packet and authorize the external action.

WebMCP Leverage

WebMCP is the product boundary, not an integration badge. CommandCanvas registers a concise Site Tools catalog with document.modelContext.registerTool(...). A supported ChatGPT host can read the current canvas and selection, create or update semantic objects, organize work, use shared history, transform selected material, prepare a packet, and request a staged send.

Each tool has a strict schema, read/write annotations, cancellation support, lifecycle cleanup, phase requirements, role checks, revision guards, and compact results. Registration can remain stable or change by phase behind one feature flag, while execute-time authorization stays identical in both modes. The agent never receives camera frames, database credentials, the user's ChatGPT credential, or authority to approve an email send.

WebMCP makes CommandCanvas meaningfully better because selection, object identity, room revision, collaborator activity, packet approval, and receipts are live application context. ChatGPT acts against the same objects the participants can see and manipulate rather than a detached export.

Execution

CommandCanvas uses Next.js 16, React 19, TypeScript, Zustand, Zod, and a custom DOM/SVG spatial canvas. Pointer, touch, stylus, voice, and hand input converge on one intent model. Stable object changes use expected revisions; high-frequency cursor and movement previews remain ephemeral.

Supabase Postgres stores rooms, objects, receipts, packet snapshots, and send requests. Supabase Presence represents connected participants. Broadcast carries cursors, revision notices, and participant-bound small-room WebRTC signaling. The no-signup judge preview creates a bounded temporary authenticated identity only after entry; durable rooms use six-digit email OTP.

Browser hand input uses local MediaPipe landmarks in a worker and keeps frames on the device. It implements index-finger drawing, pinch-to-grab, two-hand resize and canvas zoom, open-palm pan, minimize, recoverable side-edge trash, and visible interaction states. Pointer, touch, keyboard, typed command, and voice equivalents remain available.

Optional in-page Live Voice uses gpt-realtime-2.1 through a user-owned API project key and a narrower capability catalog. The production release completed a real controlled provider chain from spoken transcription through a create_board function call, visible board creation, and a voice receipt. A separate production provider test rasterized a sketch to PNG, sent it to an image-capable OpenAI model, validated the structured result, and preserved the source beside the generated diagram.

The current release also passed 1,723 automated tests across 164 files, responsive production acceptance at nine viewport sizes, two independent browser contexts sharing Supabase Presence, cursors, durable mutations, and receipts, controlled browser-to-browser WebRTC media, and native Chrome 153 WebMCP discovery, lifecycle, and cancellation.

Potential Impact

The gap between discussion and follow-through appears in planning, teaching, design review, research, technical interviews, and team decision-making. CommandCanvas turns live spatial work into reusable objects while the context and participants are still present. Rough notes can become decisions and actions. A sketch can become a chart or diagram. The room can become a reviewable packet.

Because the system is domain-neutral, it can support a student drawing a study model, a team mapping a launch, or an analyst sketching a chart.

Creativity & Ambition

The memorable interaction is not gesture control or generation alone. It is a shared semantic world in which a person can draw with an index finger, explain the idea aloud, receive a structured result, physically reposition it, watch a collaborator modify nearby work, and let ChatGPT continue through explicit capabilities. Human, collaborator, voice, and WebMCP actions appear in one receipt stream.

Preserving the rough source matters. It makes the transformation understandable, inspectable, and reversible instead of hiding the human contribution behind a generated replacement.

Challenges and lessons

The hardest problem was maintaining one trustworthy state model across fluid previews, durable collaboration, agent tools, undo, and consequential external actions. Spatial input became more reliable when gestures produced the same small intents as a pointer instead of owning separate business logic. Agent actions became safer when registration was treated as discoverability, while every execution still rechecked current room authority and state.

We also learned to separate evidence boundaries. A green source test does not prove a physical hand, real email delivery, or a ChatGPT-hosted Site Tool call. Native Chrome 153 has verified the deployed registration and lifecycle behavior; the exact ChatGPT built-in-browser invocation and final named-device hand rehearsal remain required recording gates and will not be claimed from other evidence.

Try it

Open the temporary no-signup judge preview at https://commandcanvas.vercel.app/demo. It needs no signup, login form, password, third-party account, API key, or configuration for the core canvas path. The public preview intentionally reports packet delivery as preview-only rather than claiming an email was sent.

Built With

Share this project:

Updates