guit.AR

Inspiration

Initially planning on creating something in the field of gaming, the team received an email from one of Microsoft's Tech Evangelists about hackers who might be interested in creating VR/AR applications for HackMIT 2017. Being awestruck by the amazing piece of hardware that Hololens is, the team immediately voted to go for it!

At first glance, the team knew they had to create something fun and educational that would be used intensively by adults and children alike using a novel product.

What it does

guit.AR is a mixed reality application running on Microsoft's Hololens. On the front end, the app boasts functionalities of being used for learning to play the guitar by the absolute newbies, or for advancing skills by testing out their abilities by the intermediate and the pros. Using complex image processing frameworks, the app reads music files and translates it to corresponding staff notation. It also comprises of a reliable Text-To-Speech system which renders all the musical notes for the user's given song choice and puts it right in front of them—thereby helping them learn.

How we built it

Using Unity3d's unique dev environment for the Microsoft Hololens, along with the Mixed Reality toolkit by MSFT, the team built an intuitive and fun user interface for easy access. The server was hosted on a local machine running python scripts, to which the Hololens would wirelessly connect and transmit/receive data.

Challenges we ran into

There were multiple challenges that the team ran into:

Unable to run the Mixed Reality Toolkit due to mismatch of versions in Unity.
Hololens does not have an in-built synth, which did not allow us to use midi files.

Accomplishments that we're proud of

The biggest accomplishment that the team members are proud of is connecting something that has been there for ages with something that is all about the future.

What we learned

Always know what you're getting into. Using a newly released gadget might sound fancy but it comes with a downside—that of there not being enough tutorials available online for development.