Deep Dreaming Music Generation
Leyton Ho, Livia Zhu, Kevin Cox
Introduction
Could Deep Dream be applied to music? In our project, we classify songs as a certain music genre and modify them to fit a different genre in a Deep Dream-esque generation. We do this to see if Deep Dream can be expanded to the area of audio and to understand how our music genre classifier interprets music.
Methodology
We used the Free Music Archive dataset as it contains a vast selection of songs and associated metadata, including the genre that the song falls under.
Our project has 3 components:
- Autoencoder: We created a convolutional autoencoder to reduce the inputted songs to latent feature vectors and trained it using the MSE between the input and output audio vectors.
- Feed-forward Genre Classifier: We trained a classification NN using the genre labels from the dataset.
- Genre Switcher: After fixing the weights for the autoencoder and classifier, we take in a song and treat as a
tf.Variable, then calculate the cross-entropy loss between the desired classification and the actual classification to run gradient descent.
Challenges
We ran into a couple of main challenges during our project. An initial challenge we ran into was figuring out the best way to represent audio to train our models. We eventually settled on vectors of the amplitude at each time t in a time series, limiting the audios to lengths of 5s and with a lower sampling rate using the librosa library. However, we also considered .wav and .midi files, and those formats could be interesting to experiment on in the future.
While we were designing our model architecture and training our models, we also ran into the challenge of processing the dimensionality of the data. There were 55,125 amplitude time stamps for each song. This posed interesting questions on how to best reduce the dimensionality while preserving key features. We experimented with approaches like convolutional layers for the autoencoder and different dense layers for the classifier, but acknowledge that this is still an open problem.
Finally, we had some trouble getting comprehensible audio output from the deep dream model — it often sounds like static. This is likely related to the previous challenge, since the deep dream model relies heavily on the autoencoder. Part of the challenge here was that there weren’t conventional examples to go off of beyond the concepts of deep dream described in blogs like this. As we were switching to a new domain (music rather than images) without a strong blueprint, this posed a major challenge.
Results
We had two major results: an accuracy metric for the autoencoder to genre classifier pipeline and subjectively listening to if the genre switcher produced reasonable output.
It’s important to note that both the classifier and the genre switcher were dependent on the output of the autoencoder. We saw good results when testing the autoencoder on its own. We were able to reduce the dimensionality of the latent space to a fifth of the input space and the decoded output sounded very similar to the original input.
The feed-forward classifier used the output of the autoencoder to label each song with a genre. Our classifier was able to classify songs with a top-3 accuracy of 77%. We decided to use the top-3 accuracy for two main reasons. First, identifying genres based on amplitude values (the form of input data we used) seemed like a bit of a fickle task because dramatically different songs can have similar amplitude values at certain time stamps. Second, there is inherent similarity between musical genres (i.e. instrumental and experimental).
Our genre switcher unfortunately only produced static. We think it may have overfitted to the classifier and not correctly distinguished features of the songs found in each genre. The utility of a latent vector of amplitude values as properly representing the important parts of an input song is also an open question — we think that information loss plays a large part in the static output.
Reflection
From the beginning of our project, we knew that our goals were somewhat ambitious, as it involved implementing our own idea of switching music genres using a method which we had not previously seen in research papers. It also involved synthesizing 3 different models together in order to achieve this goal, with training on datasets larger than anything we had previously trained on during class projects. In terms of our base, target, and stretch goals, we met the base for classifier accuracy of 70% for top 3 accuracy, and we reached our base goal for the generator of producing a novel input. Our approach ended up changing over time, as it started off with only basic feed forward networks. However, we added convolution to the autoencoder to account for temporal dependencies of music. This was the largest pivot that we made during the project.
If we were to continue work on this project in the future, there are several things that we could do to improve performance. First, instead of the input being the amplitude of sound waves at various timesteps, it would probably be more interpretable for the computer to use a fourier transform of the input to train, because they have more discernible patterns than the overall sound wave. Second, improving the autoencoder would likely result in significant improvements to the genre switching, as the encoding is the basis of the classification. This could be improved by experimenting with different architectures, like something with more layers or different types of pooling. The last change that could make a significant difference is using a discriminator that determines whether certain songs are legitimate or created by our model. This could be added to the loss of our generator and result in more realistic, less staticky output. Our takeaways for the project were synthesizing many of the concepts we learned during class in building the autoencoder and classifier models, learning how to handle audio input in models, and learning how to design a model which changed the inputs instead of the model weights.
Log in or sign up for Devpost to join the conversation.