What if you could hear spatial relationships and see acoustic harmonies?
Musical theory, like so many other domains has suffered under the constraint of 2 Dimensional visualization.
No longer is that the case.
What it does
This application is a musical thinking tool. It remaps our spatial sense to our auditory sense. We can now hear space and see harmony; Your position X (left,right), Y (up, down) and Z (forward, back) become pitch-shifting controllers of 3 separate musical notes. This means that every location in space is a "possibly-harmonic" location. Like pokemon floating in special places, this world has orbs that show you the exact locations of Major Chords, Minor Chords, and more. Never-before have we had the accessible, 3D space-tracking and augmented reality capabilities the Hololens provides. We exploit that power to visualize harmony, and auralize space.
How We built it:
We used the Hololens' spatial/location mapping abilities to know where our user is in 3D space. We then got a hold of our user's X,Y, and Z coordinates, using each to modify the pitch of a sustained musical note-- one for each note of a three-part musical chord respectively. We then placed small orbs in the environment at X,Y,Z positions the correspond to musical harmonies (we chose points whose coordinates create a major chord). We used Unity, C#, musical frequency data crunching, Hololens SDK, and Redbull.
Challenges I ran into:
The main challenge we ran into was the fact that Hololens starts the global coordinate system origin (0,0,0) at the viewer's head. We really wanted it at the viewers feet. People are different sizes ... I'm sure people who know the device know how to solve this, but it was our first time behind the wheel of the Hololens.
Accomplishments that We're proud of:
Getting people to roll around and jump around, explore, and discover a whole new relationship with sound, music and harmony.
What We learned:
What's next for Harmony Space
Performing arts Posture, pose, positional music mod: Data auralization