StrumAlign

Inspiration

Many beginner guitar players struggle with proper technique and current guitar apps only rely on audio signals to provide training while YouTube videos don't provide real-time feedback which is why we wanted to create an app that uses computer vision to provide real-time video-based feedback for users.

What it does

The application uses computer vision to map out the user's fingers on a guitar's fretboard and performs curvature analysis on their fingers and determine if their fingers are curved enough and that their wrist is properly straight.

How we built it

We used openCV and MediaPipe for hand tracking and then utilized vector math to calculate the angle of each finger based on 3 joints located within the finger. Finally, we utilized React to create a frontend to integrate our computer vision finger tracking.

Challenges we ran into

Some challenges we ran into were integrating the frontend with the computer vision models and displaying them on a website with a nice user interface

Accomplishments that we're proud of

We're proud of the fact that we were able to get a working project with so many complicated features to integrate properly into a nice fleshed-out product.

What we learned

We learned a lot about our tech stack like React, openCV, and integrating backend and frontend plus also using authentication with Firebase.

What's next for StrumAlign

We wish to make this more widely available by scaling up to cloud platforms and adding more functionalities such as chord detection and musical tempo training as well.

Built With

• expo.io
• firebase
• flask
• javascript
• mediapipe
• numpy
• opencv
• python
• react