Have you ever tried learning the piano but never had the time to do so? Do you want to play your favorite songs right away without any prior experience? This is the app for you. We required something like this even for our personal use. It helps an immense lot in building muscle memory to play the piano for any song you can find on Youtube directly.
What it does
Use AR to project your favorite digitally-recorded piano video song from Youtube over your piano to learn by following along.
Best Practicable/Scalable, Music Tech
How we built it
Using Unity, C# and the EasyAR SDK for Augmented Reality
Challenges we ran into
- Some Youtube URLs had cipher signatures and could not be loaded asynchronously directly
- App crashes before submission, fixed barely on time
Accomplishments that we're proud of
We built a fully functioning and user friendly application that perfectly maps the AR video onto your piano, with perfect calibration. It turned out to be a lot better than we expected. We can use it for our own personal use to learn and master the piano.
What we learned
- Creating an AR app from scratch in Unity with no prior experience
- Handling asynchronously loading of Youtube videos and using YoutubeExplode
- Different material properties, video cipher signatures and various new components!
- Developing a touch gesture based control system for calibration
What's next for PianoTunesAR
- Tunes List to save the songs you have played before by name into a playlist so that you do not need to copy URLs every time
- Machine learning based AR projection onto piano with automatic calibration and marker-less spatial/world mapping
- All supported Youtube URLs as well as an Upload your own Video feature
- AR Cam tutorial UI and icons/UI improvements
- iOS version