Chord Hero

Inspiration

There are many hobbies that people enjoy to do and create happiness. Music is one such hobby, and the guitar is a main aspect of music that is commonly learned and played by many. The issue with hobbies such as this is the happiness does not last as people grow bored with hobbies such as playing the guitar. Our goal is to bring back the motivation and encourage those to continue to play the guitar that gives them happiness by providing a more engaging and interactive app. This app will be different from others as it will both help you improve through visual and audio monitors, but it will also allow you to create the music instead of just copying motions.

What it does

Our app will be set-up as a game which will record visually and through audio the user playing the guitar. As you select a song, you will be shown a screen that will show the beat of a song and encourage the user to correctly hold and play the guitar. As you correctly play a chord on beat, you will receive points. You will lose points if you play a chord incorrectly by holding the guitar wrong or have the wrong sound. As you progress, you will be able to achieve high scores and learn how to play your favorite songs on the guitar.

How we built it

Finetuned a YOLO model paired with mediapipe to generate a key-point detection over a person to detect which chord is being played, paired with a CNN model on mel-spectogram feactures from the audio data. The game itself is built on pygame.

Challenges we ran into

Getting good datasets and merging them, improving the accuracy, training a model on data in a limited time frame.

Accomplishments that we're proud of

Training the visual model to help pickup the chords

What we learned

mediapipe, pygame, sklearn, torchaudio, cv2, pillow

What's next for Chord Hero

Future improvements to Chord Hero involve the improvement of the visual system to ensure that it can pick up all chords on the guitar that you may want to play. With the introduction to more chords, this will allow for more songs and more complexity to appear and encourage guitar players to continue to push to improve their abilities so that they can continue to have fun playing.

Built With

• cv2
• mediapipe
• pillow
• pygame
• sklearn
• torchaudio