Why we built it
Since the onset of the COVID-19 pandemic, musicians, music educators, and music lovers all over the world have needed to distance themselves from their fellow creators, making the sharing of new musical ideas and compositions difficult. While many excellent desktop software packages exist for single-user music composition Finale, Sibelius, etc. music collaboration is hard online, and doesn't allow for the real-time edit tracking experience users have come to expect for document-based work on platforms like Google Docs. That's why we built ScoreShare, to help fill the need for an online music score editor to fit the modern musician's needs.
What it does
ScoreShare allows users to create music scores right in their web browser, edit score properties, notes, and rhythms all using an easy-to-use ABC musical notation, and share them with others while seeing changes synced in real-time.
ScoreShare also allows for users to import and export any ABC musical notation from another program, allowing composers and music students to start in any program they want, bring it into ScoreShare for collaborative work, then export it back to their local program of choice.
How we built it
React.js for our front-end framework, Node.js and Express to run our API backend and WebSocket server, abc.js for musical score rendering.
By using Material UI with the React front end, it allowed us to create an intuitive user interface that can be accessed from any device and any browser. The front end was then deployed via Google App Engine, so that anyone can access it from anywhere without the need to manually scale and configure the runtime environments.
By using Express and Node.js for our back end, as well as MongoDB for our database, we were able to create a robust containerized server capable of managing as many connections as well need by deploying it out to Google Kubernetes Engine. This tech stack combination also allowed us to easily break our service into two main services (the webapp and the server), both of which work entirely independently and connect via REST endpoints and the WebSocket protocal.
Challenges we ran into
The biggest challenges were trying to maintain the correct order of edits by users even when network latency caused edit conflict race conditions, and needing to entirely change the music rendering library and approach to generating and interacting with music part-way through the hackathon.
To solve the order-of-edits problem, we modified the "edit distance" dynamic programming algorithm to account for the current position of the mouse cursor to restrict legal edits, then used that to backtrack to identify exactly which characters were changed, and how (insert, delete, replace). This worked a lot better than our original idea of working with keystrokes directly to record changes, so that users can interact with a normal textbox with all the expected features (click to move the cursor, drag to select, copy/paste, etc.) while the back end websocket server views all of the data entirely as incremental differences that get stacked on top of each other based on their time stack to provide the final view. This means that we don't need to worry about latency or race conditions between multiple users on the real time editor.
We also built the entire tech stack from the ground up without using any preconfigured devops tools like Firebase, so that was also a new challenge for us, although because we were building our own WebSocket server, most of our team's typical tools (serverless Cloud Functions, Firestore, etc.) aren't really suited for that use anyway.
Accomplishments that we're proud of
We’re proud of the usefulness of the idea, the technical challenges overcome, and hope that it can empower musicians and music lovers distanced by COVID-19 all over the world to work together to safely bring their collaborative ideas to life.
By the end of the hackathon, we got our Minimum Viable product working (the real time editor with the sheet music renderer), so we're happy with that!
What we learned
Through this process we learned a lot more about working with the WebSocket protocol, ABC musical notation, React hooks, and client-side music rendering.
What's next for ScoreShare
In the future, we would love to expand the musical editing capabilities to include a drag-n-drop score editor interface, add to the list of score interaction features, include more options for music playback, and add more expressive and intuitive graphics for multi-user cursors during collaborative editing. We'd also like to transform it from just a collaboration server to a long-term editing server, with users, permissions, and a file structure (think like Google Drive for sheet music).
Try It Yourself!
- Clone the repo at https://github.com/HarrisonLavins/ScoreShare
npm installfrom the "server" folder
npm run start:devfrom the "server" folder
npm installfrom the "webapp" folder
npm run startfrom the "webapp" folder
- Open out a copy (or more) of
localhost:3000and try it out! Feel free to use the console and websocket debugger to view how the edit distance works as well!