Inspiration

As children, we all wished a picture in a book could come alive: the dog jumps off the page, the bird flies out of the frame, the painting turns into a world you can walk into. Mixed reality headsets can finally make that wish real, in your own room. Life in 3D, started with one question: what if every ordinary 2D picture were a doorway?

What it does

Life in 3D turns printed (or virtual) pictures into living, interactive 3D experiences on Meta Quest 3. Look at a picture, show a reverse victory sign, and what's inside steps out into your room:

  • Buster the drone flies out, and you pilot it with hand gestures. Double-tap any surface with your index finger to land it there, or take on the Ring Challenge.
  • Robin the bird flies around the room, perches on your finger and plays an insect hunt and a feeding game. Robin and Buster can even duel each other.
  • A dog and a cat jump out and follow you around. They respond to a clap, a come here beckon, a sit sign and pointing; the dog plays fetch and the cat plays with a toy mouse and yarn. Both have real barks and meows that never repeat the same sound twice in a row.
  • A marble maze that you tilt with one or two hands, holding it like a real wooden board.
  • A hamster maze adventure where you steer the hamster with tiny thumb microgestures: swipe to walk, tap to jump. The hamster collects food, jumps over fires and can climb onto your palm.
  • An interactive painting: pull the cottage out of the canvas, splash the waterfall, then make the OK sign with both hands and pull them apart to open a portal and step inside the painting's valley. You can change the time of day, weather and season from a menu on your palm.
  • A smart plant lesson that teaches photosynthesis with hands-on visualizations: grab the sun, water the pot, look at a leaf up close, then take a quiz.

A shaka sign sends everyone back into their pictures.

No printer? No problem. The Virtual Pictures mode hangs all eight pictures on your real walls (or floating in front of you if no wall is found). Look at one and pinch to select it, and the experience starts exactly as it would with a printed picture.

How we built it

  • Engine: Unity 6 with URP, running natively on Quest 3 via OpenXR and the Meta XR SDK.
  • Picture recognition: The headset's Passthrough Camera API feeds frames to OpenCV. We match ORB features against each reference picture, verify the match with a RANSAC homography, and recover the picture's 3D pose with solvePnP, which minimizes the reprojection error:

Challenges we ran into

  • Your own hand hides the picture. Raising your hand to make the sign covers the picture you're looking at. We had to keep a picture "remembered" for a moment after it disappears from view.
  • Gestures in real life are messy. Folded fingers make hand tracking lose confidence for a few frames, which kept resetting gestures. We made every gesture tolerate short tracking dropouts.
  • Pets that wouldn't walk. A saved room scan from a different room made the pets think there was no floor anywhere. They now only trust a scan of the room you're actually in, and fall back gracefully when sensing is noisy, without ever walking through walls.
  • Performance on a standalone headset. Image recognition, animated pets, particle effects and a full 3D valley all share one mobile chip. We spread heavy work over several frames and preload models before they're needed.

Accomplishments that we're proud of

  • The moment a printed dog jumps off the paper and trots after you never gets old.
  • Walking into a painting through a portal you opened with your own hands.
  • A whole app controlled by natural hand gestures, with no controllers needed.

What we learned

  • Mixed reality lives or dies on forgiveness: real hands, real lighting and real rooms are never perfect, so every interaction needs tolerance built in.
  • Computer vision fundamentals (feature matching, homographies, PnP) are still the backbone of convincing AR.
  • Sound, timing and small animations make characters feel alive far more than polygon count does.

What's next for Life in 3D

  • Let users bring their own pictures to life.
  • More educational pictures: the solar system, the human heart, the water cycle.

Built With

  • android
  • augmented-reality
  • c#
  • depth-api
  • hand-tracking
  • meta-interaction-sdk
  • meta-quest-3
  • meta-xr-sdk
  • microgestures
  • mixed-reality
  • mixed-reality-utility-kit
  • opencv
  • opencv-for-unity
  • openxr
  • passthrough-camera-api
  • shaders
  • unity
Share this project:

Updates

Submission history