We really wanted to build something that combined music and a form of visual art. We initially liked the idea of creating a Theremin with an iPhone camera, and our idea developed into something much more creative.
What it does
iTheremin uses the iPhone's built in camera in order to detect winks and smiles. When detecting either one of those things, it plays a soft synth sound accordingly and plays a small animation. We then integrated a leap motion to detect two hands over it, in order to increase the pace of the background music.
How we built it
We used Swift in order to develop the iOS app. The backend uses a PHP API in order to send and receive data from the leap motion, written in Java. We utilised Photoshop in order to create all of the assets for the project, including the animations. We sourced some sounds online and edited them to a useable point and made them fit together nicely.
Challenges we ran into
Our real problem with this hack was developing the idea to a point that we were happy with our own skills in order to develop. Creating a Theremin using an iPhone camera would have proven to be incredibly difficult, so we adapted the idea to something we were happier with. Not only did we like the new idea more but we felt it pushed our current knowledge and would teach us new things along the way.
Accomplishments that we're proud of
We're proud of a few specific accomplishments. At first we didn't know how to do live-video processing, and we were debating for the first few hours if it was a feasible concept. When we achieved it, we were so happy. Secondly, we're really proud of how it all comes together to give a really functional interaction. It's so cool to be able to interact with something without even touching it. Getting to grips with the leap motion for the first time is also really cool.
What we learned
Animation controls with Swift. Sound controls with Swift. Using a leap motion. Using the leap motion with our own PHP API. Creating short animations with Photoshop. The power of friendship. The memes Sam will produce when unattended like a child at an airport. The limit of patience. Also how to use Airplay on a Mac.
What's next for iTheremin
We'll make it for Android and put it on the Play Store for our friends to use.