MamaCOCO

Inspiration

Flat-pack furniture comes with a manual that has no words, just drawings. People can't tell which part is which, or what it should look like from where they're standing. We all have a stool or shelf with one screw left over. The official fix is paying someone to build it for you. We wanted the manual to show you instead of making you guess.

What it does

ManualXR shows a see-through "ghost" of the finished furniture at real size, right where it will stand in your room. Then it walks you through it:

Each step lights up the part you need in orange and animates where it goes.
When you finish a step, that part turns solid.
The original paper-manual drawing for the same step shows beside it, so you can compare.

Where it runs:

Meta Quest: mixed reality in your real room. Grab, turn and zoom the model with your hands or controllers.
Android phone: camera AR. Point at the floor and tap "Place here".
Laptop: a 3D preview for testing (not for building).

AI features (Claude vision):

Find your product: take a photo of the manual's front page and the AI works out which product it is.
Plain-language steps: the AI reads each step's drawing and writes it in plain words, with the screws (code and count), tools and warnings.
AI animation plan: the AI decides which parts go on in each step. The app can play that plan and score it against the real step order from our dataset.
Read any manual: for products we don't have in 3D, photos of the pages still become a written step guide.

How we built it

One Unity project that picks Quest, phone or laptop mode when it starts. Quest uses OpenXR with Meta's OpenXR features; the phone uses ARCore for floor detection.
3D parts and step order come from [dataset name], an open dataset (CC-BY-4.0). It currently covers 36 products.
AI server written in [language], calling Claude's vision model. Answers are cached, so a second request is instant and works offline after the first run.
Accuracy test: one key (F9) runs the AI on all 36 products and saves the scores to a spreadsheet.

Challenges we ran into

Making one app work across a headset, a phone and a laptop, each with different controls.
Turning wordless drawings into clear text without the AI guessing too much.
Being honest about the AI's limits: it can explain any manual, but making 3D parts from a manual alone is still a research problem.
Getting the model to show up reliably in phone AR is still being fixed.

Accomplishments that we're proud of

Seeing the finished item at real size in your own room before you start.
An AI feature we can actually measure: on our 36-product test, the AI got [score].
One project running on three kinds of devices.

What we learned

How to place 3D content on real floors and tables in mixed reality and phone AR, how to use AI vision on drawings with no words, and why testing AI against known answers matters more than a good-looking demo.

What's next for ManualXR

Fix phone AR, add more products, track the real parts in your hands so steps complete by themselves, add a "Stuck?" photo check, and research building 3D guides straight from any manual.

ManualXR is not affiliated with IKEA. 3D data from dataset name.