Hand Sounds

Inspiration

How can bodily presence expand in the immersive - exceeding our physical human form? Embodied multisensory interactions lead to a new perception of the self and the space around us. Space becomes us, we become space. Experience becomes presence, presence becomes the experience.

What it does

This audio-visual AR work uses hand tracking - the gestures generate and alter unique audio stems that engineer the visual profile of the experience through interactive VFX and shaders. The user can calibrate the HCI interface through recording custom gestures that the system stores as a velocity profile, that later on feeds into the interaction system. The result is a unique-to-user, customized audio-visual journey.

How we built it

We use MRTK and deployed in Hololens 2 and Magic Leap. We created the sounds using MPC Live and Splice. The art, 3D, shaders and digital assets were done using Adobe Medium, Maya3D, Blender, Unity VFX, Shader Graph, and hand painted imagery. Everything was combined and came to live using Unity. We recorded the video thought Miracast. We also implemented Interactive Machine Learning (InteractML) that would have enabled the user to train the experience real time with live recorded itnervals of bodily gestures that then triggers the interactions (and the data is stored locally in one single file the user own). The training is controlled in the Unity node interface, we could not pair this up to a UI on the Hololens.

Challenges we ran into

Setting up the devices and trying to combine Hololens with Magic Leap specific MRTK libraries. Transforming gestures into data that we could interpret and translate into music and shapes.

Accomplishments that we're proud of

To make everything work rhythmically and seamless. To be able t develop the experience in a way that it engaged you into trying to explore more and more.

What we learned

To combine all of our skills into this amazing experience and convey that to a great number of people. We learned a lot about how Hololens keeps track of the meshes it sees through its "scene understanding". We discovered new types of shaders and materials and ways to make them audioreactive.

What's next for Hand Sounds

The sky is the limit. There are many possibilities to further extend this experience and even target different areas besides just art or entertainment.

Built With

• adobe-medium
• magic-leap
• microsoft-hololens
• miracast
• mpc-live
• mrtk
• splice
• unity