Inspiration
We all are passionate about traditional Chinese musical instruments. However, we cannot access it due to the different obstacles caused by the pandemic. Thus we developed the inspiration that we want to build "instruments" which can exceed time, money, and location to bring entertainment, inspiration, and some sense of "home" to all the passionate Chinese traditional music lovers like us.
What it does
Our program allows users to play traditional Chinese music on major pentatonic scales like a virtual piano.
How we built it
Applying the open cv and google media pipe modules, we realize the AI hand recognition and calculate the landmarks per frame. we design a one-to-one mapping between notes in the major pentatonic scale and customized gestures so that it can play traditional Chinese music. We managed to collect more than 15 thousand pieces of data in our training set to best classify and predict the hand gestures. In the meanwhile, the musical sound is modified and designed, to reach a futuristic sonic ambiance.
Challenges we ran into
We ran into the question of interaction between different methods. Since we created multiple python files and methods, we ran into conflicts and mismatching between the series of programs.
Accomplishments that we're proud of
Our model has an average accuracy above 85% and it can classify and predict hand gestures very well.
What we learned
We learned how to adjust our algorithm to better train our model so that it can best classify and predict hand gestures.
What's next for Somatosensory Instruments
We will add more instrument support and make it web accessible for everyone.
Log in or sign up for Devpost to join the conversation.