• 

  Playing along!

• 

  Enter a YouTube URL and launch AR camera after getting our marker

• 

  Building muscle memory!

• 

  Intro

Inspiration

Have you ever tried learning the piano but never had the time to do so? Do you want to play your favorite songs right away without any prior experience? This is the app for you. We required something like this even for our personal use. It helps an immense lot in building muscle memory to play the piano for any song you can find on Youtube directly.

What it does

Use AR to project your favorite digitally-recorded piano video song from Youtube over your piano to learn by following along.

Category

Best Practicable/Scalable, Music Tech

How we built it

Using Unity, C# and the EasyAR SDK for Augmented Reality

Challenges we ran into

• Some Youtube URLs had cipher signatures and could not be loaded asynchronously directly
• App crashes before submission, fixed barely on time

Accomplishments that we're proud of

We built a fully functioning and user friendly application that perfectly maps the AR video onto your piano, with perfect calibration. It turned out to be a lot better than we expected. We can use it for our own personal use to learn and master the piano.

What we learned

• Creating an AR app from scratch in Unity with no prior experience
• Handling asynchronously loading of Youtube videos and using YoutubeExplode
• Different material properties, video cipher signatures and various new components!
• Developing a touch gesture based control system for calibration

What's next for PianoTunesAR

• Tunes List to save the songs you have played before by name into a playlist so that you do not need to copy URLs every time
• Machine learning based AR projection onto piano with automatic calibration and marker-less spatial/world mapping
• All supported Youtube URLs as well as an Upload your own Video feature
• AR Cam tutorial UI and icons/UI improvements
• iOS version

APK Releases

https://github.com/hamzafarooq009/PianoTunesAR-Wallifornia-Hackathon/releases/tag/APK

Built With

• augmented-reality
• c#
• easyar
• unity