Hummingbird

• 

  Swipe to see our prototype webapp!

• 

  Record your humming anywhere, anytime!

• 

  Our program will extend your 5-10 seconds hum into minutes long melodies. Select the part you like to proceed.

• 

  Share your amazing work to your friends on socials, on just throw the melody back into our program to genarate a even cooler song!

Inspiration

We wanted to make music creation more accessible for the masses. We saw a YouTube video from CodeParade where he uses neural networks, specifically an autoencoder, to generate 16 bar songs. His implementation works on a user-end by controlling 40 different sliders that modify 40 different latent space variables picked by the neural network to produce a song. We thought that this implementation might still be unintuitive for an untrained user, as they would need to have some concept of what each variable means, and it would require still quite a lot of trial and error. We decided that a better way to make song creation then, was to produce a program that could work from a hum, instead of from latent space variables.

What it does

Our program takes as input short wav file of a hummed tune. It then takes this hummed tune and converts it into a midi file. It then takes this midi file, and expands it to try and build a whole song that encompasses at the least a very similar vibe to the original wav file.

How we built it

We built the program by stitching together two pre-trained neural network models. We split the task into two main sections: first, taking the hummed tune and performing pitch detection on it to produce an appropriate MIDI file, then secondly, taking the produced MIDI file and expanding it into a one to two minute long song. For the first section, we used SPICE, a neural network trained by TensorFlow which is available on the TensorFlow Hub (this can be found at link). For the second, we used figaro, and their pre-trained model (link).

Challenges we ran into

Our first challenge came during the first model - we found that on one of the audio clips that we inputted the output midi file was completely different to the input. We did some testing with other audio files and found that the issue was two-fold: for one, the audio clip is somebody going "la" instead of humming, which lead to pitch correction issues, and secondly, the audio clip was really fast, with not much diction to define when one note ends and another starts. We found that this wasn't an issue when consecutive notes were different, but when the same the model counts both notes as one long note. We are currently running into some challenges artificially extending the length of our songs. We found that when you input a short clip into the second file you often get a short clip back, so we're looking into ways to appropriately expand the first clip to get a decent output from the second.

Accomplishments that we're proud of

We manually reconfigured the weighting for the notes on the first model, so that we could appropriately account for faster songs. The original confidence threshold meant that a lot of notes would get dropped, and so we had to test a few different confidence values to find one that would not drop too many notes whilst also not introducing too many incorrect notes either.

What we learned

We learned a lot about transformer models for neural networks, as we found that the autoencoder model wasn't appropriate for the kind of product we were hoping to build. We also, in that same vein, learned a lot about autoencoders to be able to make that call. This showed us how transformer models could be used to generate music. This also gave us experience working with TensorFlow.

What's next for Hummingbird

Potentially training our own models for the MIDI expansion segment. We had attempted to do this during the hackathon but were limited by hardware available to us. This forced us to use somebody else's pre-trained model, that wasn't quite appropriate for the use case that we had in mind. We also wanted to implement a webapp for the product to be used through, but we ran into time constraints. We've attached below some images of what the webapp interface might look like. Interpolation is the next step in terms of the actual music generation. We would like to start with a hum, and then produce a waveform that the user can see on the site. The user would then be able to highlight sections of the waveform that they would like to keep by dragging their mouse along it. The rest of the waveform would then get replaced with blank space, and interpolation would be used to fill in these gaps on the next pass.

Built With

• css3
• html5
• python
• tensorflow