Inspiration
GuitarZeno was inspired heavily by the game GuitarHero, which turned playing the Guitar into a fun and interactive experience. We were moved to creating an accessible, interactive guitar learning platform that combines hardware and software to provide a more engaging experience. Our vision was to create a platform that would be tailored to each person's individual skill level.
What it does
Using a combination of a visual detection model using OpenCV and a touch sensitive hardware breadbox device, we created an interactive platform to learn the guitar fundamentals with lessons. It provides real-time feedback for chords, strumming patterns, and song playthroughs.
How we built it
For the hardware components, we used an ESP 32 microcontroller and connected capacitive touch sensors. Alongside the hardware input, we incorporated a visual detection model using OpenCV to monitor strumming motions, with up strum and down strum being differentiated with a combination of thumb retracted or extended and moving our hand upwards or downwards. We then connected these components to get readings and feed that information into a backend FastAPI endpoint that hosts GET requests. We then developed a frontend which interacts with the backend to display lessons and feedback in real time.
Challenges we ran into
One of our main challenges included achieving precise sensor calibration and synchronizing with the visual detection model. There was a lot of latency delays between the front end as compared to the back end, which made real life sound translation an issue.
Accomplishments that we're proud of
We were able to create a working hardware prototype that integrates with the software platform to detect and respond to user input. We created a structured lesson plan framework that guides users step-by-step through learning chords and strumming techniques. By integrating our OpenRouter API, we were able to create a chord playthrough with user inputted songs.
What we learned
We learned how virtual environments function. Along with this, we also gained a deeper understanding of visual detection through our strumming mechanism and how it integrated with the output from the breadbox hardware.
What's next for GuitarZeno
Our future steps include expanding the lesson library with more advanced progression, improving the accuracy and response latency of the breadbox hardware and OpenCV detection, and further integrating machine learning models for personalized interactive lessons.
Log in or sign up for Devpost to join the conversation.