MusicBot by Producer 404

Team Member

Haoxue Wu (hwu47), Wenjun Ma (wma20), Xingyan Wang (xwang204)

Introduction

Our project is aiming to implement a music generator that could generate midi music segments of different genres after training with different datasets of midi music files, which is a structured prediction problem. We are inspired by a blog, which introduced a jazz music generator that trains LSTM RNN to generate midi jazz music files. We choose this topic because in common understanding, music is a form of human emotional expression. We want to see what a neural network without emotion can do in this field.

Related Works

In the blog mentioned in Intro, the author preprocessed the input midi file using Music21 library and trained a two layer LSTM RNN to generate midi jazz music files. After more research, we found several related works that generate music segments using different neural network structures, which are listed below. In another open source project, MusicAutoBot, the model is based on several different variations of the transformer (TransformerXL, SequenceToSequence, and BERT). Transformers produce great results on language models. But music flow does not really depend on the early notes as language does. So we might only try to use Transformers if RNN model works and time allows.

Links to Related Works

Blog: Generating Jazz Music with an LSTM Recurrent Neural Network, Riley Wong Github
Blog: Creating a Pop Music Generator with the Transformer, Andrew Shaw Github
Project Webpage: MusicAutoBot
Blog: How to Generate Music using a LSTM Neural Network in Keras, Sigurður Skúli Github
Project Webpage: Deepjazz, Ji-Sung Kim Github
Project Webpage: magenta-Music Transformer, Google AI Github
Github Repo: midi-track-merge, Star Brilliant Github

Data

We plan to use the Final Fantasy data set in one of the related works, Classical Piano Composer, to train and test our model for the first phase. This dataset contains 93 segments of music. It's about 746 KB in total. For the second phase, we will collect open source midi files of different genres available online for further training and testing. For examples, piano sheets in classical music.

Methodology

Model Architecture

The architecture of our model will include an embedding layer, one or two LSTM layers, three dense layers and corresponding activation layers. The input tensor will first be embedded into the embedding matrices which will be fed into the LSTM layers following by dense layers and activation layers. At the output stage, the results from model will be transformed back to midi file.

Challenges

We think the hardest part could be preprocess. It takes some background knowledges about chord progression to transform midi files into training data that is compatible with RNN and Transformers. Also, the evaluation of the model could be a problem as how good is a piece of music is something very subjective. Whether the output is a good prediction can vary from person to person.

Possible Next Step

If the LSTM RNN model can work well, we plan to apply Transformer to train the dataset. Comparison can be done between two different models to see which one works better with music pieces.

Metrics

How to Evaluate?

Judgement on music is very subjective and depends a lot on personal preference. We are considering to use chord progressions to evaluate the “accuracy” of our model. As every song has certain chords progression, we can compare the chord progression of the output with the the chord progression of the original song to see whether the model outputs reasonable melody.

Goals

Base Goal: Train and test our model with Final Fantasy dataset.
Target Goal: Train and test our model with other midi music datasets of different genres.
Stretch Goal: Apply Transformer model to our datasets.

Ethics

As the dataset contains more and more sources of music, the model will be more and more powerful. If the result of a music generator is too human-like, it may be a threat to the position of human musicians. Music is a really subjective idea. The difference between auto-generated music and music that is carefully composed by humans may become insignificant coming into people’s preference. This might hurt the human musicians’ interests because the resources put into these two kinds of music can vary in amount while the market value of them can be really similar.
As mentioned above, music can be really subjective. Meanwhile, it can also be objective coming into ideas and underlying patterns used by musicians. This kind of pattern can certainly be learnt by applying deep learning. This will lead to another question. Will auto-generated music which uses someone’s music as dataset be considered as plagiarism or infringement?