LiveCanvas

Inspiration

Creative product meetings contain more useful context than their notes preserve. A designer says "make this card purple." A founder says "keep this number exactly as it is." A developer says "move this above that." Everyone in the call understands because they can see the screen and the pointing. The transcript alone does not.

The result is a familiar handoff tax: the meeting ends, someone reconstructs the references, creates tickets, and later produces the visual iteration the team actually wanted to react to.

We built LiveCanvas around one conviction: creative meetings should not end with notes; they should end with a new version.

What it does

LiveCanvas is a collaborative visual workspace with a Meet style call, a selectable shared canvas, and one deliberate generation action: Show me.

While people talk, LiveCanvas keeps one synchronized context clock for:

• who spoke;
• what they said;
• when they said it;
• which screen, slide, or page was active;
• which exact registered component they clicked or pointed at;
• what that component looked like at that moment; and
• which ideas were accepted, rejected, corrected, superseded, or protected.

The AI does not edit after every sentence. The team can brainstorm and disagree naturally. When someone presses Show me, LiveCanvas sends one consolidated evidence window to GPT-5.6. GPT-5.6 turns the messy conversation into structured decisions and allowlisted UI operations. Zod validates the response, and a deterministic mutation engine applies only safe changes. The result appears immediately with an explanation, supporting evidence, version history, compare, restore, and Undo.

This makes deictic language useful instead of ambiguous. "Make this red" is not a generic prompt: it remains attached to the component clicked during that sentence, on the screen active at that timestamp.

PowerPoint is the demo, not the product boundary

We chose a PowerPoint review for the video because slides make the core innovation easy to see in under three minutes. Native slide and object IDs give the audience an intuitive proof that "this" on slide two stays bound to that exact object even after the team moves to another slide.

LiveCanvas itself is a canvas agnostic meeting to version protocol. The same component registry, synchronized evidence, intent compiler, mutation engine, and version model already power the included analytics dashboard and landing page. The architecture is designed for interfaces, dashboards, reports, prototypes, whiteboards, and author approved code backed previews any visual product that can expose stable component metadata and safe editable properties.

The current MVP is honest about its boundary: imported PowerPoint, the built in deck, dashboard, and landing page are AI mutable declarative canvases. HTML/ZIP projects can be inspected through a passive sanitized preview, but LiveCanvas does not yet rewrite arbitrary repositories, execute generated code, or deploy software.

How we built it

The application uses strict TypeScript, React, a Next.js App Router compatible architecture, Tailwind CSS, Zod, the official OpenAI JavaScript SDK, Vitest, and Playwright. It is deployed on Google Cloud Run with separate web and WebSocket relay services plus TURN fallback for restrictive networks.

Every selectable object belongs to a component registry containing a stable ID, human label, project/page context, parent relationship, editable property allowlist, current values, and rendering reference. For PowerPoint, a bounded browser local OOXML parser extracts supported slides, text, shapes, geometry, styles, and raster images while rejecting macros, external active content, unsafe paths, DTD/entities, and malformed packages.

The meeting layer uses WebRTC for live media and a validated WebSocket room protocol for presence, shared timestamps, transcripts, cursors, clicks, navigation, canvas snapshots, generation locks, restore, and Undo. Camera and microphone remain independent from evidence capture. One participant pressing Start working opens the shared context window; every participating browser can then contribute its own finalized speech and exact component interactions. Raw audio and camera frames never enter the GPT-5.6 request.

How GPT-5.6 is used

GPT-5.6 is the semantic compiler between human discussion and a small deterministic UI language.

On Show me, it receives the current registry and canvas state, participants, ordered transcript and interaction events since the previous successful version, event time page/slide context, component snapshots, the previous version summary, and the allowed mutation types.

It returns Structured Outputs containing:

• accepted, rejected, superseded, or ambiguous decisions;
• exact target component IDs;
• allowlisted operations;
• reasons and confidence;
• evidence event IDs; and
• clarification requests when the evidence is insufficient.

GPT-5.6 decides what the team meant. Application code decides whether and how that intent may change the canvas. Unsupported operations, unknown targets, non editable properties, missing evidence, arbitrary HTML, and executable code are rejected.

How Codex accelerated development

Codex was our product engineering partner across the full build, not a one shot code generator. It helped turn the product brief into a vertical slice; design the event, registry, intent, mutation, and version contracts; implement the Meet style room, WebRTC negotiation, relay protocol, PowerPoint parser, canvases, evidence UI, and deterministic demo; research the current GPT-5.6 Structured Outputs integration; and repeatedly run, debug, and harden the project.

The most valuable acceleration came from closing the loop. Codex did not stop when a screen rendered: it added semantic fixtures, protocol tests, hostile import cases, multi browser media tests, hosted smoke tests, production build checks, deployment documentation, limitations, and this submission package. The final gate includes 117 passing unit tests and 22 passing end to end journeys.

Challenges

Grounding "this" without guessing

The difficult problem was not changing a color. It was proving which component "this" meant while several people, screens, clicks, corrections, and cursors were interleaved. Stable IDs, snapshots, one room clock, and evidence linked operations made that ambiguity testable.

Preserving discussion instead of keyword matching

Real meetings contain proposals, rejections, reversals, and protected elements. The prompt and validation layer preserve the entire decision history while preferring the latest explicit agreement and returning clarification instead of inventing a target.

Keeping generation safe and reversible

Arbitrary code generation would weaken validation and Undo. A deliberately small mutation language required more schema work, but it made every result explainable, testable, and reversible.

Making a live demo reliable

Browser speech recognition, device permissions, model latency, and WebRTC networks vary. We separated media from context capture, added typed instructions, built deterministic Demo Mode using the same contracts, added TURN fallback, and tested real multi peer synchronization in production.

Accomplishments that we're proud of

• A complete, polished product flow from lobby and pre join through meeting, contextual review, generation, explanation, and versioning.
• Exact participant-, timestamp-, page-, and component-grounded evidence.
• One consolidated GPT-5.6 request instead of continuous editing.
• Safe Structured Outputs plus a deterministic mutation engine.
• Real room links, three person media, shared voice/click context, cursor presence, late join, generated canvas synchronization, and Undo.
• A real macro free PowerPoint adapter with separately selectable objects and downloadable generated state.
• Dashboard and landing page canvases proving the concept is broader than slides.
• Demo Mode that works without permissions or credentials while exercising the production contracts.
• A deployed GCP instance, documented architecture and limitations, and comprehensive automated QA.

What we learned

The most valuable AI context is often not more text; it is the small piece of synchronized application state that makes the text unambiguous. We also learned that explicit generation moments improve creative collaboration: the team can think aloud without the canvas thrashing after every sentence, then ask for one coherent version when the decision is ready.

What's next for LiveCanvas

Next we would add higher quality streaming transcription, authenticated persistent workspaces, larger room media infrastructure, durable branches and named checkpoints, stronger PowerPoint fidelity, design token aware responsive canvases, and reviewed adapters for Figma and code backed previews. Those integrations would preserve the same principle: AI proposes structured intent, deterministic product code applies it, and humans can always see the evidence and Undo.

Creative meetings shouldn't end with notes. They should end with a new version.

Built With

• codex
• google-cloud-run
• gpt-5.6
• next.js
• openai-api
• playwright
• react
• tailwind-css
• typescript
• vitest
• webrtc
• websocket
• zod