We have been musicians and programmers both for many years, and we work with MIDI projects often. I play piano; I had the idea of doing a sightreading app that would force you to not look at the piano keyboard while playing, and that could actually track what you were playing. The latter seemed obvious: MIDI. However, a seemingly simple solution the first issue would be a VR headset for a mobile device. The player would get the sheet music generated for them where they can clearly read it, they would not be able to look at the keys, but they would get reasonable feedback on accuracy. Thus, Piandroid.
What it does
It (hopefully by the submission time... Whoops.) will
- automatically generate sightreading sheet music based on rules of music composition
- render the sheet music in VR in a Google Daydream View headset through an Android .apk
- connect to a piano keyboard through MIDI (particularly for the demo, over USB, but in theory over Bluetooth as well) and score the player's performance based on the differences between generated music and played music.
How we built it
It uses C# and Unity for rendering and generating sheet music. Generation of sheet music is in three stages: chord progression generation by taking random walks following good composition practices, rhythm progression generation by random walks over pregenerated rhythm patterns of certain lengths, and melody generation by keeping within a key signature and generated chord and rhythm progressions. Unity handled the VR rendering, and MIDI code is based on open source Android demo code for MIDI testing.
Challenges we ran into
Unity code is in C# for rendering and generating sheet music, but all MIDI interfacing is in native Android Java code. It's great. We love it. It took us 6 hours to compile and get a working Hello world program that connects the two.
Accomplishments that we're proud of
The melody generation is quite musical for being rather random and very quickly coded, a subject that has been a topic of research for almost 100 years. It also compiles, which is a feat in and of itself.
What we learned
Android MIDI is not ready for the prime time.