CamPiano

Inspiration

Thinh's girlfriend tricked him into thinking this would be an easy project and that she would love him if he made something for her to compose as she does not have a piano keyboard and composing on laptop is annoying. I don't know how I got here.

What it does

It uses a convolutional neural network to detect hand movements and see when keys are pressed, then after exiting, converts the music played to a .mid file for future use.

How we built it

We built it mostly using OpenCV with Python, finding tutorials and documentation online to support us.

Challenges we ran into

The first problem was tracking 2 hands. We ended up rewriting the main software for actually running the hand detection. The second problem was that we could not simultaneously play sounds and display the video at the same time. We tried multithreading, scamp, winsound, musicpy, and many other libraries to fix this. We moved from using .wav files to just using .mid files. Eventually we settled on pygame due to it being able to play sounds without freezing the video display. Then, we had the problem of playing 2 or more notes at once. This was eventually fixed with pygame as well. After that, we had the problem of playing notes with both hands which we used a lot of logic to work out. Making it feel and look somewhat like actually playing a piano was also very difficult. Plus the hand sign detection was a little finicky.

Accomplishments that we're proud of

I did not expect us to actually be able to finish the project since it felt very challenging, but we actually finished our project.

What we learned

We learned how to do our first project, how to use OpenCV, and how to use Git among other things. We had a lot of fun doing it with our very professional solutions to bugs.