Music Transcriber

Inspiration

Music Ideation We realized through personal experience that the best ideas for melodies, songs or beats often come outside of the studio. As a result, many creative artists are unable to capture their idea in the moment, and by the time they get around the equipment they need, that idea is often gone. Therefore, we decided to create a mobile music translator.

What it does

Mobile Music Translator Application Our product is an app that allows you to humm or vocalize a recording and then translate that recording into a MIDI (Musical Instrument Digital Interface). You can then assign an instrument to this MIDI and use your resulting music for personal or professional reasons. We are planning to add to this MVP later on to allow you to use an LLM to add instruments to or modify your music.

How we built it

Building Process We first identified a model that could help us turn humming/vocals into a MIDI (Spotify's Basic Pitch: link). We pre and post processed the audio inputs and outputs for this model to achieve better performance, particularly dealing with the presence of background noise. We then build a front-end and back-end software stack to create our user application. We build the front-end using React Native and the back-end using Flask to load the model. We also used Expo GO as a framework to run our application.

Challenges we ran into

Main Challenges We ran into challenges in two main fields: the model and the application. With the model, Basic Pitch did not have the accuracy we desired straight out of the box. Additionally, because our audio is coming from users, it was difficult to get consistent clean audio inputs. Dealing with variations such as different voices, microphone qualities and background noise all made implementing the model more difficult.

From an application perspective, it was difficult to call the model from front end, connect the front and back ends, and scale the application with more recording data due to our reliance on multiple APIs (pretty-MIDI, labrosa, fluid-synth, Basic Pitch).

Accomplishments that we're proud of

Our Accomplishments We are proud of building a fully function application that can take user input and turn it into a successful MIDI/allow the user to choose any of our four instruments for that MIDI. Having a fully functioning stack in 36 hours is something we are proud of.

What we learned

Learnings We were able to gain experience working with various music APIs. We also learned how to turn MIDIs into an actual instrument output.

What's next for Music Translator

Looking Ahead We plan to add LLM capabilities to modify/add to your music as well as change instruments, layer instruments, change genre etc. This will add a ton of customization and creative agency for our customers beyond what Music Translator currently offers.

Built With

• basic-pitch
• expo.io
• flask
• fluid-synth
• labrosa
• pretty-midi
• python
• react-native
• spotify
• typescript