cover
3D view 1
Phone controller UI
3D view 2
Gallery
House by the River (sketch)
The Starry Night

Inspiration

This year's SBUHacks 2026 theme, Steel and Steam, got us thinking about museums. A museum visit is mostly staring: you stand behind a rope, read a small label and move on. You can't walk into a painting or interact with it. We wanted art to be something you explore, not just look at.

What it does

Canvas Express turns drawings and paintings into a 3D world you explore with your phone as the controller.

Scan the QR code on the TV, pick a colour, and walk your own figure through the artwork. No app needed.
Gemini reads the picture and writes a description of everything in it.
ElevenLabs is your custom tour guide: it reads each description aloud and gives every object its own sound.
"Step inside" turns your phone into a 3D view of the artwork's world, with the main objects as real 3D models.
Snowflake saves every scene, so any visit can be reopened from the gallery.

How we built it

Front end: JavaScript, Three.js and custom GLSL shaders, built with Vite. The TV shows the artwork as a 2.5D depth stage, and the phone builds a full 3D world from it.
Phones as controllers: a Vercel WebSocket function, with Upstash Redis passing messages between servers. Each TV gets its own private room.
Gemini returns the whole scene as structured JSON: each object's name, outline, depth, motion, sound idea and description.
Depth: Depth Anything V2 runs right in the browser (Transformers.js on WebGPU), so measuring depth costs nothing.
3D models: Tripo turns the main objects into textured 3D models.
ElevenLabs: text-to-sound for the effects and text-to-speech for the tour guide's voice.
Snowflake saves each project as JSON through its SQL REST API.
Vercel hosts the pages and serverless functions, and Vercel Blob stores the images, sounds and models. API keys never leave the server.
Claude Code was our pair programmer for planning, code and tests.

Challenges we ran into

Gemini can't generate 3D objects. We split the job: Gemini reads the picture, a depth model in the browser measures distance, Tripo makes the 3D models, and we build the world ourselves.
Planning the layout and UI: the TV, the phone controller and the 3D view had to feel like one product and be readable from across a room.
Keeping many phones in sync on serverless: connections close every 5 minutes and land on different servers, so we reconnect before closing and route every message through Redis.
Our first 3D models came out completely black. We fixed it by drawing them unlit, with their own textures.

Accomplishments that we're proud of

Successful 3D generation: one photo becomes a world you can step inside on your phone.
A beautiful UI/UX design, with a train-journey theme: a Conductor, tickets and a gallery.
A stranger can join from a QR code in seconds, with no app and no account.
Gemini, ElevenLabs, Snowflake, Vercel and Tripo all working together live in one app.

What we learned

How to make AI a dependable part of an app: ask for structured JSON, check every answer, and plan what happens when it fails.
Real-time multiplayer on serverless needs a message bus and one place that owns the state; for us, that's the TV.
Use each AI tool for what it's best at, instead of asking one to do everything.
Plan for things breaking during a live demo, so every service has a backup.

What's next for Canvas Express

Bringing this technology to engineering and architecture: turn a machine sketch into something that moves, or a floor plan into a building you can walk through.
A curator mode, so museums can check and correct descriptions before visitors arrive.