Guit.AR was inspired by lack of technological advances in the fields of arts and culture. In the past few decades the digital revolution resulted in many innovations that helped students learn faster. But the field of music was not able to adopt these changes because learning music is a very different process than other fields. The pandemic made distant learning necessary making learning music difficult. Guit.AR is an attempt at allowing distant learning for learning guitar.
What it does
Guit.AR is an AR based webapp which teaches users guitar by projecting the strings to be stuck for playing a song directly on to user's screen so that all they have to do is to pluck the highlighted strings to play the song.
How we built it
The project uses Aruco.js which is a famous CV library for marker detection. It also uses React.Js for the frontend, MongoDB for database, Express.js for backend and Node.js for hosting the backend.
Challenges we ran into
Detecting the fretboard was the toughest part and the accuracy of the projections depend on it. The strings of guitar are very close together and require very high accuracy to trace them correctly. We initially tried to use neural networks for this purpose but it was unable to get required accuracy especially due to lack of large enough dataset. Due to this we had to improvise and use marker based AR for detecting the fretboard posed multiple algorithmic challenges but in the end we were able to get a solution which resulted in very high accuracy.
Accomplishments that we're proud of
We are proud of the alogorithm that maps the projections with respect to marker position. It results in very high framerates while giving very high accuracy.
What we learned
Time management and improvisation were our biggest learnings from this build. Our initial plan for building it didn't workout so we had to figure out another solution and implement it within the time limit.
What's next for Guit.AR - Guitar Simplified!
We want to extend Guit.AR from guitar to other instruments as well. This will be especially helpful for protecting cultures which are getting extinct.