Sublime

Sublime is how we designers and creatives will be prototyping worlds. By mixing AR, image and world models, users can draw what’s on their minds. See them in 3D, immerse themselves in their world.

Comment

Inspiration

We kept seeing the same gap in the "AI world" wave: text-to-world tools are magical but abstract: you type words and hope. Meanwhile the most natural creative act on earth is picking up a mar ker and drawing something. We wanted the shortest possible path from a scribble on a piece of p aper to a real 3D thing you can walk around inside. No modeling software, no app install, no login — just draw, point your phone, and step into it.

What it does

Sublime turns a marker drawing on paper into an explorable 3D world, entirely in your phone's br owser.

  1. Draw any object on a sheet of paper with a marker.
  2. Point your phone at it. Sublime detects the page in real time, hugs its edges with an on- screen outline, and captures a deskewed, cropped scan of your sketch.
  3. Watch it come to life. An AI render "develops" over your sketch, turning the flat drawing into a realistic image.
  4. It becomes 3D. That image is turned into a textured 3D model that stands on the physical page in AR, at paper scale, and tracks the paper as you move the phone around it. Scale it up, restyle it.
  5. Step inside. Describe an environment in a sentence and the table falls away — you're stan ding in a generated World Labs Marble world with your object living inside it. Tap the object an d it comes alive.

The whole thing runs in a mobile web browser. No install.

How we built it

Client (Vite + TypeScript + three.js), running on the phone:

  • Paper detection in a Web Worker using OpenCV.js — RGBA→HSV segmentation, contour finding, and 4-gon fitting at ~30 detections/sec on a downscaled frame.
  • Planar pose solved from scratch in JS: a DLT homography decomposed against assumed camera intrinsics (stock OpenCV.js ships no solvePnP), round-trip-verified to ~1e-16.
  • Smooth AR via a transparent WebGL layer over the camera feed, with the projection matched to the camera intrinsics so models land pixel-exact on the page. Detection only sets a target; a 60fps loop lerps/slerps toward it, so tracking stays buttery regardless of detection rate.
  • Gaussian-splat worlds rendered with Spark, with the Tripo GLB comp ene.

Server (Node + Express) orchestrating three paid generative APIs:

  • Google Gemini — sketch → realistic image render.
  • Tripo — image → textured 3D GLB (plus retexture for restyling).
  • World Labs Marble — a text prompt → a full explorable gaussian-spl
  • A content-hash disk cache means re-scanning the same drawing costs zero credits — essentia l for rehearsing a live demo.

Windows-to-phone testing was done over a cloudflared tunnel so the pho TTPS dev server directly.

Challenges we ran into

  • Paper detection on a real desk. Our first detector thresholded on brightness — which compl etely failed on a light wood desk where the white paper is nearly as bri "paper" quad ballooned to the whole tabletop. The fix was to segment on paper-ness, not bright ness: a saturation mask (neutral paper vs. chromatic wood) AND a value table), which are complementary across surfaces. That one insight — it was a detection problem, not a warp problem — unblocked the entire scan step.
  • No solvePnP. Stock OpenCV.js doesn't include it, so we hand-rolled the homography solve and pose decomposition in JavaScript and validated it against synthetic
  • Tracking jitter. Raw per-frame detection is noisy; the target-plus-lerp/slerp render loop was what made the model feel glued to the page instead of vibrating.
  • Model size vs. speed. A 15MB GLB took 8+ seconds just to download on the phone — we tuned Tripo's face_limit and tier to get a model visible in seconds.
  • The "toy from above" look. A world generated from an overhead phone shot looks like a dior ama. We added a virtual eye-level camera you place in AR to bake a prope nt around the object.

Accomplishments that we're proud of

  • A complete sketch → realistic render → 3D model → explorable world pipeline that runs on a phone in the browser, with no app install.
  • Real-time paper tracking with a from-scratch planar pose solver — no black-box AR SDK.
  • Chaining three separate generative AI services (Gemini, Tripo, Mar that feels like one seamless piece of magic.
  • The model literally stands on your physical drawing at the right s e — the moment that makes people go "wait, what."

What we learned

  • Gaussian splatting and the Spark renderer — a genuinely new way to represent and stream 3D sce nes to phones.
  • The World Labs Marble text-to-world API, and how to composite classic mesh geometry (Tripo GLB s) inside a splat world.
  • Hard-won computer-vision intuition: the right feature to segment on matters far more than th e algorithm downstream — and the smoothing layer between raw detection a eel" actually lives.
  • How to design an integration contract so a two-person team can build t pipeline side fully independently against mocks, and integrate by flipping a flag instead of wr iting glue at 4am.

What's next for Sublime

  • Multiple drawings on one sheet, each becoming its own tracked mode
  • In-AR camera baking to author richer, more coherent first-person worlds around any object.
  • Save, re-place, and share — a library of your creations you can dr nd shareable world links.
  • Tighter object/world coherence (lighting, scale, grounding) and a nati tracking.
  • Multiplayer: two people drawing into the same shared world.

Built With

Share this project:

Updates