Inspiration

Since the COVID-19 pandemic and resulting quarantine forced students and teachers to begin learning remotely, and trying to play music over video conferencing software can be both laggy and a terrible experience for your ears, we began searching for ways to stream music in MIDI form to amend this.

What it does

Keyshare resolves problems of latency, bandwidth, and audio quality by allowing student and teacher to connect on browser and stream MIDI data directly from the student's computer (via connected MIDI instrument) to the teacher's computer.

How we built it

Keyshare's frontend is built in Javascript with React and managed by Node.js/npm. Although mainly centered around the material-ui framework, various other packages were crucial to the process, including - but not limited to - react-piano (which we wrote custom JS using low-level WebMIDI for), sound-font-player, lodash, and midimessage.

Keyshare uses WebRTC to transmit MIDI data between connected clients. Although a Firebase signaling server is used to negotiate the initial handshake between clients, after this handshake, all communication becomes peer-to-peer, preventing server-associated performance costs.

Challenges we ran into

Perhaps the greatest challenge we experienced was that of networking. Our relative inexperience at it and our troubles with the world of asynchrony made for difficult times. Beyond that, we also experienced troubles working with MIDI implementations, mostly due to the fact that WebMIDI features are experimental and as such its API is relatively poorly-documented, and also owing to the need for low-level modification in one of the MIDI-related Node packages (see above, in the 'how we built it' section.)

Accomplishments that we're proud of

The fact that we managed to achieve all of our goals and more and surmount all of the challenges we ran into (see above) is perhaps the greatest accomplishment we could hope for. Seeing our finished product is what makes us the proudest.

What we learned

Through this project we continued solidifying our knowledge of technologies we were comfortable with, such as front-end dev in React and Node, while at the same time experimenting with technologies - and even concepts - we had little-to-no experience with, like WebRTC-based networking, WebMIDI, and Firebase. We learned many valuable lessons (for example, networking is nothing to be scoffed at) and came out far better attuned to modern web development and better equipped to handle its problems.

What's next for Keyshare

Although we managed to accomplish all of our must-have features, there are also some nice-to-have features that would be nice to implement in the future. Some features that may be implemented in the future include a masterclass mode that would support multiple listeners rather than just one-on-one sessions, live chord identification to aid musical teaching, and greater freedom of sound through greater variety of instrument sounds and the ability to choose a specific MIDI input.

Note: The YouTube video attached is a re-upload of the original submission video, and contains minor clarifications at 2:33. We confirmed with the organizers that the re-upload is acceptable. The original submission video can be found here: https://www.youtube.com/watch?v=PtERKFCXCg0

Share this project:

Updates