There were several ideas that we had, for example we wanted to do, such as an arduino pingpong picking robot, however we wanted to do something that is much more challenging. We thought about this project because we are cs majors who are also music lovers, however there is difficulty for beginners to learn piano, which is caused by not knowing where the keys are or the pitches for the music. In addition, we wanted to also do a web-based AR which is based out of an API which is released not very long.
What it does
It is an Augmented Reality web-app (supporting various devices including google daydreamer) which teaches the user how to play a music piece by guiding them with a virtual view of the key that should be played and tells if they are wrong, ending with a recommendation on how they could improve.
How we built it
We used AR.JS which is a very new js library built for ar/mr application, and we used a electric keyboard as input device and midi files as user input music files. We parsed the music input files which are time based inputs to AR-based 3d renderings. To top that we integrated a webapp, which is hosted on Google Cloud so the other people could use it in the future.
Challenges we ran into
First, AR.JS is very new, so it nearly has no documentation, and we had to work out by trial and error. In addition, we had to get music into the correct format and timing, which was a serious issue.
Accomplishments that we're proud of
We did not sleep a lot, but we made our first ar/vr app and it is also the first in AR.js. :) We also can do something that is visually cool :)
What we learned
A lot :). Tons of debuging:/
What's next for Augmented Studio: Your personal Virtual Piano Teacher
Make it visually more appealing, adding more sound effect, give better recommendations.