-
-
Michael rests his hands on the "keyboard".
-
A stunning rendition of the masterpiece "Happy Birthday" is performed.
-
Developing Invistrument.
Inspiration
The mean global per-capita household yearly income is $2,920 dollars. Today, grand pianos go for about $25,000, and even lower-end upright compact pianos are around $3,000. They are, to put it bluntly, completely unaffordable. If the average person didn’t spend any money at all on food, housing, utilities, clothing, literally anything, for an entire year, they would still be unable to afford a cheap piano. Even in developed countries such as the United Kingdom, Japan, and Finland, negative average household savings rates ensure that the overwhelmingly vast majority of people will never be able to afford to learn to play the piano. The data undoubtedly shows that learning the piano is simply completely inaccessible to the average person. This is absurd. Research has consistently shown that playing music lowers blood pressure, reduces stress and anxiety levels, strengthens immunological responses, and with sufficient practice provides resistance to mind-related diseases such as Alzheimer’s disease and dementia. It just isn’t right to deny these easily achievable incredible benefits to most of the world.
What it does
Invistrument employs augmented reality to display a virtual piano through the phone camera onto a tabletop. Users tap their fingers onto the tabletop to simulate piano playing, and Invistrument plays the corresponding notes.
How we built it
Our app is built with React, and certain site display elements such as the sheet music are built on OpenSheetMusicDisplay, VexFlow, and MusicXML. We utilized Tensorflow to detect the users hands as well as the positioning of their fingers. We were able to use the AR.js library to access augmented reality. The 3D model of the piano was created in Blender. Most of the project is written in javascript.
Challenges we ran into
We leveraged quite a few libraries for this project, and we quickly found out that different libraries don’t necessarily mesh together well. We had to figure out how each library worked in order to implement them for our purposes. Certain features required old versions of libraries, which caused conflicts within the library and required us to trade off features that we wanted to add to Invistrument.
Accomplishments that we're proud of
We’re very pleased with how our UX turned out, as well as the ability to navigate the menus by pinching one’s fingers. It’s an interactive interface that we feel meshes very well with the application. In addition, we were able to wrestle Git into submission much more effectively this year than in previous years, by being much more careful with changes and strategically coordinating work.
What we learned
Our initial vision for the project was much larger, but we quickly learned that prioritizing a minimum viable product before achieving anything else was crucial for a successful project under such a tight deadline. We compromised away a lot of bells and whistles to deliver a working keyboard, but this doesn’t mean that they won’t return in the near future.
What's next for INVISTRUMENT
In the future, we hope to further flesh out playing along with the displayed sheet music, displaying correct and incorrect keys played by the user and potentially giving the user a score or grade at the end of the song. We also hope to add the ability to virtually play more fingered instruments through Invistrument, such as the guitar and the harp. We’d like to create the ability to play these instruments through a Google Cardboard viewer, creating a fully immersive augmented reality experience.
Log in or sign up for Devpost to join the conversation.