"Stinky toed mice"-Digital costumes for the opera

Opera singers practicing with their costumes
On stage one day soon

Inspiration

I help pediatric patients to dream big and create their own cartoon music videos and video games. One day I was working with a patient who was having a rough time and in need of a positive distraction. Nothing would make her laugh, until we stumbled upon stinky toed mice. She thought the idea of mice who had stinky feet was SO funny she was laughing and soon the entire room was laughing too. So that's what her cartoon music video was about- a couple of mice with stinky toes. Now, we are bringing her cartoon to real life using professional opera singers, digital costumes, and augmented reality.

What it does

This AR experience is intended for the theater. The audience will scan their pamphlet and it will give access to the digital costumes to dress the performers. They will be in charge of putting the performers in different digital costumes during the show. Options include: fancy clothes, rotten cheese shoes and stinky green gas coming from their feet.

How we built it

We got the mice modeled and then rigged the skeleton in Blender.
Kept the various outfits and accessories as separate mesh.
Imported the files inside Lens Studio. Used the 3D Body tracking component to track the movements of the users.
Attached male mouse to first user and female mouse to second user using the body index
Used the custom segmentation technique to erase out the real body so only the mouse avatar is visible for added immersiveness.
Created a custom 2D carousel to pick between the various outfits and accessories using Scripting.
Added custom smoke VFX and modified the graph to make it look like a stinky smell coming out of the cheese shoes.
Programmed the facial expressions tracking using the API available to match the avatar's facial movements with the users'
Since while recording we wanted to disable the carousel and to avoid the hassle of keeping another render target, created an invisible trigger area on the top half of the screen to toggle between the visibility of carousel

Challenges we ran into

Main challenge was to control the expressions accurately with the users'. It took a bit of time to figure that one out properly.
Second one was modifying the VFX graph to accurately represent the stinky feet
Outfit & accessories positioning on the Rat avatar to match properly

Accomplishments that we're proud of

We are definitely proud of how it turned out and most importantly how it's being used to bring dreams to reality for children battling through tough times. We also love seeing how it is bringing joy to the community and raising awareness for pediatric cancer.
It has already crossed over 300K Impressions, 290K+ Lens Plays & shared by 3000+ people all around the world on snapchat. Feels happy to see making it such an impact on global scale and lovely to see people enjoying our creation

What we learned

Advanced body tracking with facial tracking
VFX enhancement
Custom UI programming

What's next for "Stinky toed mice"-Digital costume for the opera

Professional opera singers, a string quartet and a local high school are all learning the patient's song to create a live performance of “Stinky Toes.” The audience will participate in running the show by dressing the performers during the song. The project as a whole is a special example of how this technology can be used for good and bring communities together. The AR experience will also be used to spark innovation for the students and faculty to adapt to use digital costumes in the future productions. Not only would it save on cost of physical costumes and sets but it can bring a sense of awe and wonder to the audience. In a whole we are using AR to bring more immersion and audience participation to live action performance.