One of our members experiences a low form of synesthesia, and two of our members are Bollywood and Bhangra dancers. We wanted to combine these two art concepts--"seeing" sound and movement in a Doctor Strange-esque way. As a result, we created Synesthesia, an application that allows users to "conduct" a song by changing pitch and color, dance to the song, as well as paint an abstract visual representation of their movements.

What it does

Synesthesia is a game that combines the most popular forms of art--music and drawing--into one, allowing for an immersive and escapist experience.

How we built it

We built the application on Unity, coding in C#. In order to obtain user movement, we interfaced the Myo armbands with our Unity code, allowing us to track the player's arm movements.

Challenges we ran into

Initially, we really wanted to make Synesthesia into a VR game--imagine a fully immersive blank canvas where you can get lost in your music and art. However, we quickly realized that the Samsung Gear VR--our first choice for a VR headset, due to the mobile portability--and Android as a whole is not compatible with Unity-made applications.

We then attempted to create an Oculus Rift VR game; however, we lacked the necessary hardware to build it.

Thus, we ended up creating a desktop game that accomplishes our core, initial goals: to be able to "conduct" a song and being able to physically represent music via user movement.

Another challenge was that we were unable to build our program on a Windows device. We were unable to debug and resolve this issue, so we were only able to build a Mac osX version of our application.

However, the biggest challenge was learning Unity, as none of the four members had any prior experience. Creating a Unity application /and/ interfacing that with an external input--the Myo armband--was extremely difficult, given our inexperience.

Accomplishments that we're proud of

Our biggest accomplishment is being able to create our initial idea exactly as how we had envisioned it. Although we were unable to use VR to create a fully immersive game, we adapted quickly and managed to make our game work without any compromises (aside from the platform change away from VR), despite our lack of experience in Unity.

What we learned

We learned how to create games using Unity. We also learned how to interface a "modern" output device--the Myo armband--to our game.

What's next for Synesthesia

Polish, polish, polish. We would like to have a starting screen, as well as options to change songs. We also would like to improve our game's camerawork, as it can often be janky. In the same vein, the Myo itself can often be inaccurate--both in user tracking and initial setup. In short, better user interface and controls are greatly needed.

Our biggest update would be porting our application to VR. We feel that our game is best suited for VR, as it is meant to be fully immersive.

We would also like to support Windows devices.

Lastly, we have a lot of unnecessary assets. We would like to organize these better, as it will improve performance and decrease file size.

Built With

Share this project: