As a team consisting of a lazy musical genius and a musically incompetent teenager who both love software engineering, it is inevitable at some point that we are going to build a bot that can assist us in generating music.

What it does

So we decided to build a Messenger chatbot that can convert your personal singing voice into an accompaniment track with chords.

You record your singing as you chat with our bot. The bot's backend will analyze your vocals and generate an accompaniment track with chords to play along with your specific voice. Perfect for when you have someone who sings and someone else who wants to play along with an instrument but is not...that good to pick up the other person's vocals.

How I built it

To analyze input vocals, we built a chord progression prediction deep learning model which is very similar to the ones used to generate text given a seed (specifically, it uses a series of LSTM networks that feed into a final classifier, which assigns sub-sequences of melodies and chords with a chord that goes with the next melody note. We also used the (proprietary) Melodia algorithm to convert singing audio into midi notes.

We then wired up this analysis code into a Facebook Messenger chatbot backend webhook (built with ExpressJS) that allowed us to customize the conversation flow with our users. We hosted the webhook on Heroku and store our converted audio files on MongoDB Atlas. We had to code up the chatbot webhook ourselves with a Facebook developer account because that was the only way we could integrate it with our analysis code which involved a lot of shell scripting.

Challenges I ran into

  • We could not find a way to send midi files back through Messenger due to security reasons -> we uploaded our converted files to a third-party database (MongoDB Atlas) and returned a GET endpoint for users to get their chords file.
  • The audio to midi conversion algorithm is very finicky, and it seems that it only works well if you sing loud and clear.

Accomplishments that I'm proud of

We finished it

What I learned

  • Heroku bash system
  • Converting audio to midi is very hard
  • Shell scripts automation in NodeJS

What's next for Chordsky - Personal Accompaniment Chords Generator

  • We're thinking of incorporating the singing audio with our generated chords file.
Share this project: