SnapFit

Inspiration

Furniture manuals are the most universally hated documents on earth: exploded diagrams that assume x-ray vision, step numbers that hide which steps are actually independent, and zero feedback when you bolt the wrong leg on. Meanwhile, image-to-3D generation got good enough that a single product photo can become a segmented, part-level 3D model in minutes. We put the two together: what if every furniture listing shipped with an interactive, spatial, voice-coached assembly guide — generated automatically from the seller's photo and PDF manual?

What it does

SnapFit is a two-sided platform for 3D assembly guides.

For sellers: upload one photo (any background) and optionally the official PDF manual. A fully automated pipeline extracts the product onto a clean background, generates a part-segmented 3D model, names every part semantically, compiles a step-by-step assembly plan, and publishes the listing — no human in the loop, about 10–15 minutes end to end.

For buyers: open a listing on desktop, phone, or a headset straight from a QR code. Drag the explode slider to see the furniture fly apart, grab glowing parts and snap them into ghost outlines, and ask the built-in AI coach anything by voice — "where does this part go?" — while it highlights parts and plays placement demonstrations in the scene. On Quest, the whole thing runs in passthrough AR at 1:1 real-world scale, so you assemble the sofa in the room where it will actually live.

How we built it

3D generation: TRIPO image-to-model (v3.1) + Smart Segmentation v2 with the cleaned photo as reference image, giving semantic part boundaries with textures preserved.
Photo cleanup: gpt-image-2 lifts the furniture onto a white studio background with a strictly shape-preserving prompt — the output is the single source for 3D generation, the segmentation reference, and the listing cover.
Teaching an LLM to read a GLB: LLMs can't parse meshes, so we split the model into two channels: a geometry summary (per-part centroid/bbox/volume + a contact adjacency graph from point-cloud distance) and a color-coded three-view render with a color↔part legend. A vision model grounds "the four thin vertical things are legs" back to actual node IDs.
Assembly planning: DeepSeek takes the labeled parts, the adjacency graph, and the parsed manual text, and emits steps as a dependency DAG (four legs are parallel, not steps 3–6), plus per-part explode directions and common-mistake warnings. It also extracts the product's real dimensions from the manual, which we bake into the GLB so AR is true to scale.
Trust nothing: every LLM output passes a deterministic validator (DAG acyclicity, every part installed exactly once, exploded parts can't overlap, transforms copied from geometry ground truth, never from the model). Failures are fed back verbatim and the model retries — our test sofa passed on attempt 3.
Runtime: React Three Fiber + WebXR, one codebase from phone to Quest 3 passthrough. The coach is an event-driven agent (wrong part picked up, step timeout, voice questions) wired to ElevenLabs STT/TTS, answering with both speech and scene actions like demonstrate_step.

Challenges we ran into

2D input, 3D target: on desktop/mobile a mouse drag moves parts on a sphere around the camera, so a part could look aligned but never reach the snap radius. We added a screen-space snap assist that gently pulls the part to the target depth while dragging, plus a ray-distance fallback on release — VR interaction stays untouched.
Segmentation cut-faces: segmented parts have open backs where they touched neighbors, so clicks passed straight through them. Fix: invisible exact-fit box colliders per part, which also cut raycast cost from 150K triangles to a handful of boxes.
Hallucination containment: early runs had the LLM inventing part positions. The fix that stuck was architectural, not prompt-based: numeric fields are always overwritten from geometry truth, and the validator turns "please be careful" into hard rejections.
Real-world scale: generated models come out around a meter tall regardless of what they are. A sofa you can put on a table is a demo; a 3.9 m sofa in passthrough is a product. Extracting real dimensions from the manual and baking the scale into the GLB root fixed this for every future upload.

What we learned

Multimodal grounding beats clever prompting: giving the model evidence (colored renders + geometry stats + the actual manual) and validating deterministically got us to reliable auto-generation faster than any amount of prompt polish. And in XR, interaction details — snap assist, colliders, 1:1 scale — matter more than visual fidelity.

What's next

Real-time seller pipeline in the cloud (the worker currently runs locally), assembly analytics for sellers (which step do buyers get stuck on?), multi-user co-assembly, and PICO-native optimizations.