Scorify

This shows a sample audio file being processed by our website.
Our website outputs a MusicXML file that can be opened in a music notation editor, such as MuseScore.

Inspiration

We came up with this idea because of a dual interest in music and technology. Amanda commonly composes music, but finds it annoying to notate the composition in notation software or by hand. Our idea allows musicians to automate the process of physically writing down their music. Instead of spending time mindlessly notating a composition, musicians can now focus more on their creative process and making the actual music.

What it does

The user should create an audio file of a single-note melody. They will then upload their audio file to our website and will receive the option to download the newly created corresponding MusicXML file. This file can then be put into notation software, such as MuseScore. This allows the user to adjust or add to the score produced, as well as export it.

How we built it

We built our project within VS Code using Python and various Python libraries and tools. We used Spotify Basic Pitch (a machine-learning model) to convert the audio file to a MIDI file. MIDI files have information about the note pitch and duration. We parsed the MIDI file into separate distinct note objects, and then quantized the note lengths. After this, we used the mus21 Python library (which had elements that could be used to build a MusicXML file) and our note objects to build a complete score of the music. This was then returned to the user. Our UI interface was built using the Streamlit library, which allowed easy design for uploading and returning files.

Challenges we ran into

We had trouble formatting the UI design because the formatting was very specific, and we were learning how to build it as we went. We had trouble with the setup (setting up our environment, figuring out the layout for all files, and figuring out the dependencies with versions that were compatible with Spotify Basic Pitch). Figuring out how to remove background noise and jitter was difficult because we weren't sure how to distinguish the data from background noise when the pitches were similar. It was also difficult because sometimes Spotify Basic Pitch would incorrectly recognize the notes (ex: there were 2 repeating notes when that wasn't there in the original audio file) based on the way it divided up the notes. We had to write 2 methods to deal with the way Spotify Basic Pitch would divide the notes: an overlap method (when notes would overlap in time) and merge_split_notes (that would split up notes or delete notes based on its amplitude and differences in start times). We also tried to implement a preview feature where users would be able to preview their scores instead of inputting the XML file into MuseScore to see the score, but only the top portion of the file would be shown; thus, we ended up removing this feature for the sake of time.

Accomplishments that we're proud of

Our base idea was to get our product to recognize one note, and after we got that working, we improved upon our code to get a melody working. We were also skeptical about creating a website for users to be able to upload their audio files, as we didn't think we would have enough time, but not only were we able to create one, but we were also able to add many features like a download option, other input fields (DMB, title name, composer name) that further enhanced our product.

What we learned

We learned how to create a working pipeline using existing high-level Python packages. We specifically learned how to utilize the Python libraries Spotify Basic Pitch, mus21, and Streamlit. We also learned more about UI interface design and how to design and create a webpage. We learned about the calculations that go behind a platform that parses audio files and has to calculate pitch, beat length, and quantized notes (correcting the timing of each note to fit within standard rhythm).

What's next

Our next step would be for our program to accurately detect and notate multiple melodies or chords being played at the same time. This would be very useful for piano solos, as they usually have a base line and melody being played at the same time. We would need to do more testing on different types of instruments and vocals as well to make sure that our score stays accurate across different sound qualities. Eventually, this technology could be used for small groups, such as trios, or ensembles. We would also want to incorporate more features into our application. Useful features would include the option to preview the created score and to directly download it.