Inspiration

Ancient Chinese history is dramatic and enchanting. Since I'm a quarter Chinese and have practiced kung fu for 25 years, I wanted to tell an immersive story that blended ancient China with the martial arts. In addition, Chinese mythology and folklore are full of symbols, ghosts, creatures and demons.

Lessons Learned

My vision was to knit MR scenes between VR story scenes. These transition scenes would track the user's journey on a map etched on an oracle shell. When I tried to implement this scheme, I realized this content wasn't necessary. In other words, I spent most of the hackathon trying to do something that ultimately didn't work. After years of creating VR formats that have premiered at festivals, I've found that simpler is better for mainstream audiences. For example, if the player doesn't need to use an object for a task, a tap is easier and faster than a grab.

Prior to VR, I had extensive training and experience in theatre. A maxim in theatre: content dictates form. My focus is on using those tools - e.g., hand tracking, VR, MR, AR - that best serve the content.

What I discovered was that one of the main story scenes was far more interesting in MR than VR. Can I switch to MR seamlessly in a VR story? Can we get to the point where definitions of extended reality - VR, MR, AR - no longer matter because we're juggling them to suit the needs of content?

For me, this hackathon was proof of concept. That is, creators can switch between forms without having to announce or define it. As long as the content is persuasive, we can dance between the forms with delight. Tools fade from view while content takes center stage. When this happens, the medium has truly evolved.

The Build

I first built The Red Flute in PCVR for the Rift in Oculus Launchpad. The format sat on a shelf for years. For some reason, I looked at it again this year and wondered why I had abandoned it. The format was magical. So I adapted it for the Quest and implemented hand tracking. The VR-only version premiered at FIVARS '26 (Festival of International Virtual & Augmented Reality Stories) in June. During the festival, the producer emailed me and said: It's a hit!

For the hackathon, I used Meta's passthrough components. Because I've been working on another format in MR, I'm reasonably comfortable with Meta's MR functionality. To achieve an interesting MR scene, I tweaked a cinematic LUT from the Amplify Shader Pack and applied it to the passthrough layer. The solution was simple.

With regard to hand tracking, I used microgestures for teleporting and a palm menu. These components are provided by Meta, easily customized and work well.

Built With

Share this project:

Updates

Submission history