AirKeys

• 

  Playing the piano on AirKeys

Inspiration

We found through our experiences that starting the learn the piano can be a difficult tasks, for both space and money reasons. So our product aims to solve both those issues by having a free, open source piano that anyone can play anywhere.

What it does

Our product utilizes hand tracking and a computer webcam to detect when your hands "press" a key on our AR keyboard. When your finger presses any of the keys, it plays a distinct piano sound.

How we built it

We utilizes python 3.9 libraries such as openCV for video capturing and processing. We used legacy mediapipe to track hand motions using their ML model. For piano sounds, we used a library called fluidsynth which allowed us to integrate the notes whenever a key is pressed, while also avoiding mp3 audios.

Challenges we ran into

The hardest part that we had was trying to first work with mediapipe. Mediapipe changed a lot of its library not long ago, so we had to figure out how to utilize a legacy version while also leveraging other compatible libraries such as older versions of python and openCV. The other problem we ran into was finding compatible audio libraries for the piano sounds. One of the biggest problems we had was with the hand tracking from mediapipe. Its really good at detecting the front of the hand, but not so much the back like when playing the piano. This forced us to find workarounds and other methods of trying to optimize it to the best of our abilities.

Accomplishments that we're proud of

We had to learn a lot of new libraries that had a lot to work with. Mostly, we are proud that we accomplished the goal that we set out to do in the time frame.

What we learned

How to use various different python libraries and how to work with different version dependencies in our tech stack. We also learned to better collaborate and split a large project into smaller parts that each person can work on separately. We also greatly improved our github skills.

What's next for AirKeys

In the future, we would like the add an overlay that shows the notes a person should play to various different songs. We also would like to add different noises when a key is pressed, so the user has as many options or more then they would with a physical keyboard. We would also like the refine some of the hand tracking and add more test data to mediapipe to make it more reliable with the back of the hand tracking.

Built With

• fluidsynth
• mediapipe
• opencv
• python
• vscode