Project Story: A2UI Canvas & Medo

## Inspiration

Traditional UI/UX design tools—while incredibly powerful—still require a massive amount of manual layout, pixel-pushing, and tedious mapping. Worse, there is a historical disconnect between design and code; designers hand off flat shapes, and developers spend hours translating them into functional components.

We asked ourselves: What if the distance between an idea, a design, and production-ready code was completely erased?

We wanted to create a tool where you don’t just draw interfaces; you converse with them. A platform where developers and designers can orchestrate complex, real-code workflows simply by chatting with an LLM, visualizing element relationships intuitively on an active canvas, and interacting directly with live, underlying components. Thus, A2UI Canvas was born.

---

## What it does

A2UI Canvas is an AI-powered, node-based UI design tool that lets users generate, wireframe, and prototype high-resolution interfaces using natural language and image uploads.

• Chat to Design: Users talk to a multimodal LLM using the specialized A2UI/AGUI protocol to instantly generate UI layouts.
• Batch Image Analysis: Uploading multiple screenshots or rough sketches allows the AI to interpret layout patterns and instantly generate editable canvas components.
• A "Real Code" Canvas: Every element on the Figma-like interactive canvas isn't just a vector shape—it is a functional component backed by live, editable React code.
• Workflow Mapping: Elements can be linked visually to establish logical relations, creating a comprehensive map of how a user navigates the app.
• Pro-Grade Workspace: Features hierarchical layer trees, drag-and-drop grouping, layout alignment/distribution toolbars, custom keymaps, a live code editor, and full crosshair measurement rulers.
• Real-Time Collaboration & Portability: Powered by Supabase, multiple users can co-design with live cursor tracking, restore snapshots from automated version histories, or export their workspace instantly into a React ZIP file or a live CodeSandbox template.

---

## How we built it

Building a robust frontend design engine wrapped inside an AI loop required a modern, snappy stack:

• Frontend Workspace: Built with Next.js and Tailwind CSS to manage a highly responsive workspace. The canvas manipulation engine utilizes custom vector mathematics to calculate dynamic zoom levels, panning tracking, and node connection vectors.
• AI Engine & Protocol: Leveraged advanced multimodal LLMs communicating via the A2UI/AGUI protocol. This ensures that natural language prompts and image payloads render structured, semantic component configurations instead of unparsed, raw JSON strings.
• Backend & Real-Time Sync: Powered entirely by Supabase. We leveraged Supabase Database for custom project tables, auto-save version history snapshots, and custom templates. Real-time multi-user synchronization and live cursor tracking were implemented via Supabase Realtime Broadcast Channels utilizing a "last write wins" conflict resolution policy.
• Live Coding Environment: Integrated a lightweight, syntax-highlighted sandboxed code editor that maps directly back to the active canvas component state, enabling live hot-reloading of UI modifications.

Let the scale of our responsive canvas space be represented by a transformation matrix mapping viewport coordinates $(x_v, y_v)$ to canvas space coordinates $(x_c, y_c)$ given a zoom factor $s$ and pan offsets $(t_x, t_y)$:

$$\begin{bmatrix} x_c \ y_c \ 1 \end{bmatrix} = \begin{bmatrix} s & 0 & t_x \ 0 & s & t_y \ 0 & 0 & 1 \end{bmatrix}^{-1} \begin{bmatrix} x_v \ y_v \ 1 \end{bmatrix}$$

This mathematical tracking ensures that measurement rulers, crosshair alignment tools, and hovering distance guides remain pixel-perfect at any zoom level.

---

## Challenges we ran into

1. Streaming AI Outputs onto a Live Canvas: Standard LLM chat streams output raw tokens. Parsing uncompleted JSON tokens into a visual tree in real-time without breaking the UI layout was incredibly difficult. We had to design an isolated state-buffer that keeps components in a "loading placeholder" state until the structural tree is fully verified.
2. State Synchronization and Conflict Resolution: Managing a canvas where user changes, AI updates, layer ordering, and real-time cursor broadcasts all happen concurrently caused heavy race conditions. Optimizing broadcast packages to transmit thin deltas rather than heavy context payloads was crucial to maintain a smooth 60fps experience.
3. Live Code Translation: Ensuring that raw user edits in the code panel successfully update the canvas properties without causing infinite re-render loops or security vulnerabilities within the virtual DOM required rigorous sanitization and structural isolation.

---

## Accomplishments that we're proud of

• Seamless Multi-Modal Synthesis: We successfully managed batch visual processing. You can drop three separate dashboard screenshots into the chat tab, and the AI correctly synthesizes their shared UI logic into a single cohesive UI template on the canvas.
• Intuitive Node-Based Workflow Routing: Creating a visual system where users can draw connection lines between interactive elements to map application flows, feeling as natural as Figma or Miro but fully interactive.
• Production Readiness: The project goes beyond prototyping. Generating a clean React component ZIP file or spinning up a functional workspace directly in CodeSandbox with a single click bridges a massive developer-designer divide.

---

## What we learned

We learned that building AI tools is not just about writing great prompts; it’s about interface orchestration. Users don't want black-box text interfaces; they want visual control. Designing the symbiotic relationship between the Chat panel, the Layers tree, the Inspector values, and the live React code proved that the future of software development lies in hybrid workspaces where text commands and visual layouts work as a cohesive unit.

---

## What's next for A2UI - Medo UI designer

We are just getting started with erasing friction from software engineering. Our roadmap includes:

• True Offline Mirroring: Implementing localized storage sync utilizing IndexDB for local-first creation when network connectivity is lost.
• Advanced Code Guardrails: Integrating a lightweight linter within the live code panel to catch compilation and syntax errors before they attempt to re-render the canvas.
• Design Token Pipelines: Expanding the token system to automatically output Tailwind config files, custom CSS variables, and theme definitions for a variety of popular UI libraries.

Built With

• a2ui/agui
• and-custom-templates
• and-distance-guidelines.-*-**ai-engine-&-architecture:**-advanced-multimodal-llms-communicate-via-a-custom-**a2ui/agui-protocol**
• converting-natural-language-and-batch-screenshots-into-live-canvas-components.-*-**database-&-real-time-sync:**-**supabase**-handles-the-entire-backend-infrastructure.-**supabase-database**-persists-multiple-named-projects
• css
• editor
• figma-like-workspace-relies-on-custom-html5-canvas-and-dom-manipulation-vectors-to-handle-pixel-perfect-zooming
• llms
• multimodal
• next.js
• panning
• protocol
• pwa
• react
• real-time-tech-stack-optimized-for-canvas-rendering-and-ai-generation:-*-**frontend-&-workspace:**-**next.js**-and-**react**-for-the-application-framework
• realtime
• sandboxed
• styled-with-**tailwind-css**.-the-interactive
• supabase
• tailwind
• typescript
• version-histories