GuitarPicks

G chord
C chord
A chord
default

💡 Inspiration

There are countless of chords to remember when playing the guitar, especially when they look or sound the same! As musicians, we wanted a tool to help identify these chords. We were able to do this by creating a real time capture of the user’s finger placement on the guitar, beginners to experts can easily identify what chord they are playing on the guitar.

🌟 What it does

Our program uses computer vision to detect specific hand placements that correlate to a chord on the guitar and shows the user in real time. In order for the program to learn what finger placements equal specific chords, you are able to take a picture of the correct finger placement by pressing ‘k’ and associating it with a slot of 0-9 and labeling what chord it is.

🔨 How we built it

We used a built in computer vision program, created by Github user Nikita Kiselov, that allowed us to take multiple pictures of the specific hand gesture on the guitar and teach the program the correlating guitar chord. This program uses the Gesture Recognition Task from MediaPipe and includes a pre-existing model that was trained on set hand gestures. The model can be modified to add custom hand gestures that fit our purpose. Multiple key points of each custom hand gesture were taken and retrained with the model. Specifically, we used a real guitar and held our hands at the right chord position to get the most accurate data.

🚧 Challenges we ran into

We had trouble getting the output (live video detection) onto a web page for other feature implementations. We had a hard time identifying the chords when we did not have enough data or screenshots of the chord for the program to learn. Sometimes the program would have difficulty recognizing the hand while holding the guitar, which makes it hard to get accurate key points. With our limited knowledge on Computer Vision, it can be difficult to know what to improve on within the model that could potentially fix these problems.

🏆 Accomplishments that we're proud of

We are proud that we were able to get our project to detect basic guitar chords based on the data model. We are proud that we were able to understand the steps when it comes to computer vision and be able to execute that on to our program.

🌱 What we learned

We learned more about the computer vision process and all of its capabilities with patience, time, and a lot of data points. We also learned how difficult gesture recognition can be and how much data really goes into training a model for a complicated task such as this.

🎸 What's next for GuitarPicks

We would like to have more data that points to different chords to increase the detection accuracy and enhance the user’s experience. We would also like to move our live detection to an installable application with more features. These features could include a tuner, display the chord within the application prominently, and include guides/music sheets that someone could follow live, with it detecting if they are hitting the write chords for the song or not.