Inspiration
Stage performance is a great storytelling medium. However, it is restricted in its expressive potential, and cannot express genres like sci-fi. However, with AR devices, members in the audience can visualize all the special effects,normally found only in digital media, while enjoying a live performance. It was inspired from Harry Potter.
What it does
It spawns holograms/animations of fire, fog, lasers from a character's eyes, and many more special effects when the script calls it. We created an entire play that takes full advantage of this. Members in the audience watch the play live with a HoloLens, and see the effects in tandem with the play. The spawning of holograms is synchronized, so everybody sees the effects at the same position, at the same time.
How I built it
It was a university project. Using Mixed-Reality Toolkit, and Unity, we got the basic scene working. I also created a python script, that sent commands to all the HoloLens' at the same time to spawn the effects. Assets were created by students from Art department.
Challenges I ran into
Initially we couldn't get all the HoloLens' synchronized, but after going through all the docs, we learnt how to create our own "Anchor", and attach it to our root object. This allowed us to synchronize our hologram locations. Another pesky problem was that this synchronization didn't happen everytime. After some experimentation, we realized it was because our primary HoloLens was sending anchor location features from the spatial map that couldn't be seen by other HoloLens'. After anchoring objects in an open space, we started getting expected synchronization.
Our second problem was spawning the lasers from an actor's eyes. Since the actor's position is unreliable during a live show, we couldn't spawn a laser animation hologram whose location was known from the beginning. We had to utilize HoloLen's computer vision api, and machine learning api, to classify different actors on stage, and spawn the laser effect from his eye using a model designed after him.
Accomplishments that I'm proud of
- Learnt how spatial mapping works for the hololens
- Got all the required effects running
- was able to apply machine learning from hololen's api to get an accurate location for the laser effect
What I learned
- Spatial mapping
- Networking between different hololens'
What's next for Mixed Reality Theatre
- A effect where everything collapses into a black hole
- Deploying to 50 HoloLens' at the same time.
Log in or sign up for Devpost to join the conversation.