Piano Hero

Inspiration

Ever want to learn how to play the piano but don't have the time or money to take lessons? Tried watching online tutorials but can't keep track of both the screen and your hands? Well you're in luck - meet Piano Hero!

What it does

Piano Hero lets you experience the ruthless criticism of a teacher while sitting in the comfort of your own home. With just an iPhone, Piano Hero can turn any piano in the world into your own personal learning device. Simply launch the app, align it to middle C, then watch through your phone as Piano Hero shows you which keys need to be pressed by illuminating them right in front of your eyes. The app even listens to what you are playing and determines if you are playing the correct notes or not. No wires, no sheet music, no computers!

How we built it

We used Apple ARKit to place a virtual keyboard directly on top of the actual piano. Using the ARKit, the virtual keyboard can be resized, rotated, and shifted to allow the user to align its middle C with that of the real piano. Once the virtual keyboard is anchored, Unity controls the rest of the application. This involves features such as lighting up the keys, detecting the frequency of the played notes, and determining whether those notes were correct.

Challenges we ran into

The frequency identification aspect of Piano Hero was a task much more complex than we had initially imagined. We attempted to implement our own version of a Fast Fourier Transform but were unable to achieve the level of accuracy that we had envisioned.

Accomplishments that we're proud of

Modeling the keyboard and getting a very lifelike image of a keyboard which could be overlaid on top of the real one. We were very proud when all of the resize buttons worked and were properly synced up to movement commands.

What we learned

None of us had significant prior experience with either Unity or Apple ARKit, so we were able to learn a lot about both of these technologies (especially ARKit, which is very new). We also learned a good amount about signal processing, as we attempted to implement our own Fourier transform.

What's next for Piano Hero

There are many incomplete aspects of Piano Hero that we would like to fix in the coming days. We believe that optimizing the FFT and improving the note illumination will be fairly simple fixes once given the time and energy to accomplish them. We would also like to figure out a hardware aspect that builds off of a headset such as Google cardboard to allow more ease of use.

Built With

• apple-ar-kit
• unity