Inspiration -- Kalila's brother wanted to teach himself piano, however, most of the resources he found online were video tutorials on YouTube. We wanted to create a more immersive way to learn sheet music and play piano.

What it does -- The Major Key app provides the user an immersive learning experience focused on teaching students how to read sheet music and play piano.

How I built it -- We sent out surveys to conduct preliminary user research, created an affinity board, a user journey, wireframes, and a working prototype on Magic Leap. We included songs and exercises, an AR keyboard piano model, AR light up keys, and scrolling sheet music implemented with Blender and Unity.

Challenges we ran into -- While understanding the capabilities of the Magic Leap hardware we realized that the microphone was best suited for personal input and was not great at taking input from our surroundings. Magic Leap can currently only track three fingers while playing piano requires all 5 making finger tracking not feasible. Proper piano playing requires good wrist position which would also block view of fingers from sensors. This would require a work around using inverse kinematics. Our work around was to use a midi keyboard for the hack. Magic Leap does not accept serial input making plugging a midi keyboard into it impossible. We tried to implement a server using node.js on a computer that would send the midi data to the Magic Leap over wifi. This ended up being very problematic and we ended up just running the project within unity editor where both the magic Leap and the keyboard could work together. All pianos are also different sizes accounting for that would be difficult (we planned on adding tracking markers onto certain keys with stickers). After learning this we adapted our vision to best fit to the hardware within the time.

Accomplishments that I'm proud of -- Using various software development tools and network protocols in order take our vision on to Magic Leap's cutting edge device. Brain storming, wire framing, and prioritizing features to come up with a simple and intuitive user flow and Mixed Reality conventions to make accessing songs and practice usable.

What's next for Major Key -- Lesson structures and fun exercises, being able to use other instruments, gamification to make playing music more fun and appealing, social features including leaderboard concept to increase indulgence in playing music, partnership with local music initiatives and schools or tutors to promote more music learning, bring easy music learning to every home

Built With

Share this project: