We're passionate about music and Virtual Reality, so we decided to combine the two to create an unique immersive experience that has never been attempted. Our goal was to create a procedurally generated world out of an user selected song that pulsates and reacts to its ebbs and flows. We wanted to bring music into a new dimension, and to create an immersive and interactive experience out of it.
What it does
SoundScape is an immersive experience that builds an entire virtual, unique and interactive world using an user picked song as a reference. After the world is built, various elements in the world come alive and react to the song's beat, tempo, frequency, volume and other aspects of the music as it plays. The user can move their hands around freely, interact with randomly generated objects, and even create trailing light in a way rave enthusiasts have always dreamed of being able to do.
How we built it
We built this using the Unity3D Game Engine and Leap Motion. We developed algorithms to map beats and frequencies of user-selected songs to generate unique environments that respond to the music as well as the user's actions within its world.
Challenges we ran into
Procedural environment generation with a performance, getting audio to map to the VR space in a meaningful and beautiful way, engineering a gorgeous experience and designing fun interactions that made sense with the musical theme.
Accomplishments that we're proud of
We were able to create a very unique virtual-reality musical experience, making the best use of the latest technologies to develop a way to live, breathe and interact with audio-generated worlds.
What we learned
A lot about how music works (general theory, waveform, frequencies, beat, using the audio data), Virtual Reality development, Leap Motion integration and new creative ways to manipulate 3D environments with music.
What's next for Soundscape
We want to turn it into a multiplayer experience, improve our algorithms so we can create even more expansive worlds, and add even more novel ways for you to physically interact with sounds.