4% of the population has synesthesia, a cross-wiring of the sense. The more we understand how we function internally, we can create a better experience for our lives externally. In 'Synthesis' we invite you to explore how sound and taste can be associated with the colors and textures around you. Maybe the color blue makes one person feel sad, while calm for another. How can we explore the external world to discover our internal senses in a new way?
What it does
Synthesis is a spatial audio experience in Augmented Reality. It allows you to walk around your room in the Hololens, and gesture to "synthesize" an object, wall, floor, and more into sound interpretations of it's color and texture. Put the Hololens on, walk around and tap your environment to launch a sonic interpretation of it.
How we built it
We built our app using Unity, the Hololens Toolkit and Ableton Live.
Challenges we ran into
We faced the challenge of how to output effective color and texture data from the Hololens camera in realtime, and then match those to the produced sounds. Audio challenge: composing and mixing this way challenges the mind to let go of any preconceived ideas of what 'sounds' good. All the sounds have been created based on how each color and texture inspire the sounds. All the individual sounds were created separately from each other instead of composed together as a whole piece. 1 hour before submitting, some of the audio files 'disappeared' and had to recreate some of the sounds. We are missing modular sound synthesis, more advanced texture detection using computer vision, ambiance detection, and a better UI.
Accomplishments that we're proud of
Hololens development along with image processing, and sound designed unique audio to translate a mix of different forms of synaesthesia,
What we learned
We learned how to develop for the Hololens! Specifically, how to create a spatial audio experience, use gestures to place objects and manipulate the spatial mapping mesh. We also discovered how to analyse Hololens camera data in real-time. Lots of failed algorithms to classify colors. UI/UX in AR is incredibly challenge for both audio and image processing. New interactions are needed. Simplicity is key!
What's next for Synthesis
We hope to keep working on it and add more features to introduce more people to synaesthesia.