Inspiration

Before screens, puppetry was the media. A lamp, a cloth and a pair of hands were how a village watched a story. Our team is Indian and Chinese, and we both grew up near living shadow traditions: Tholpavakoothu from Kerala, India and Chinese piying. We wanted mixed reality to bring that intimacy back. The world’s oldest screen technology shouldn’t need a screen in front of you. It should happen on the table you’re sitting at, built with your own hands.

What it does

Table Tales is a seated, hands-only mixed-reality theatre for Meta Quest 3 and 3S.

  1. Find your table. Rest a hand flat on any real table for five seconds while a ring of light fills around it. Then drag the corners of a glowing frame to mark out your play area, and turn it to suit how you sit.
  2. Choose a story. Story scrolls lie on your table. Pick one up by its handle and pull the roller open with your other hand. The painting glows, and its pieces burst out in streams of light and land on the table as glass slides.
  3. Build the scene. Pinch a slide and place it in the miniature theatre in front of you. Each layer flies in at full size on a wall-sized stage beyond your table, sized to your room’s ceiling.
  4. Watch. When the scene is complete, the layers come together into one picture. The house lights go down (passthrough dims), bells ring, and the story plays as a layered film with surtitles.
  5. The ending. The stage dissolves into gold light. The story streams back into its scroll, and the next chapter’s scroll is restored from rust to its true colours, waiting to be told.

There is no fail state, no timer and no score. It’s a story you build and then watch, all within arm’s reach.

Our first story is Kadangot Makkam, a legend from Kerala, told in three chapters. Chapter one, The Beloved Daughter, is playable; chapters two and three wait in sealed scrolls.

How we built it

  • Unity 6 (URP) with the Meta XR All-in-One SDK, OpenXR, IL2CPP and ARM64.
  • Passthrough is part of the story: it is dark for the opening, at full brightness while you build, and dims like house lights for the show.
  • Hand tracking through the Meta Interaction SDK. Hand grabs use recorded grab poses, so a hand snaps around a scroll’s handle. The table keys are physical poke buttons. We read the hands directly to detect a palm resting flat for calibration.
  • Mixed Reality Utility Kit (MRUK) sizes the big stage to the room’s ceiling, and an optional mode snaps it to any wall in view.
  • Stories are data. Each story is one ScriptableObject holding its pieces, layer pictures, videos and timed captions. The scene holds only the rig, so a new story is a new file with no code changes.
  • Effects within the Quest budget. There is no post-processing and no HDR, so every glow comes from additive particle layers, plus a custom dissolve shader for the ending.

Challenges we ran into

  • Grabbing a scroll reliably. Our own gesture reading grabbed scrolls at odd points. Switching to Interaction SDK grabs with torch-style poses fixed it. Then the roller sat so close to the handle that hands opened scrolls by accident. We made the roller pinch-only and openable only once the handle is held.
  • Transparent video on Quest. VP8 video with alpha played solid on the headset, or the decoder refused it. We switched to H.264 with the alpha stacked below the picture, and a shader unpacks it on the GPU.
  • Untested rooms. Every table and room is different. Hand calibration plus a resizable, rotatable area means the layout fits anything from a desk to a dining table.
  • Testing hands without a headset. We built scripted test hands and an in-editor recorder, so every feature could be checked and filmed before it went on the headset.

What we learned

In mixed reality, the room is part of the design: your real table, your real ceiling, your own hands. Small, physical, tactile moments, like rolling open a scroll or setting a slide into a tiny stage, turn out to matter more than spectacle. Fine-tuning the hand feel can only happen on the headset.

What’s next

  • Chapters two and three of Kadangot Makkam.
  • More stories from shadow traditions around the world, including Chinese piying tales.
  • In the longer run, a tool to make, record and share your own puppet stories.
Share this project:

Updates

Submission history