We all have musical inclinations as well as interest in machine learning. As such, we wanted to pursue a project that would combine both of these interests!

What it does

It takes input text (poems, essays, articles, sentences) and turns it into music!

How we built it

Our product has three main parts. There is the Wix website which was built using Wix's UI and Javascript, the React-Native mobile app which was built entirely with Javascript, and a Chrome Extension which was built using HTML, CSS, and Javascript. The back-end was a Python Flask server which would call the Python scripts that trained the data and determines the tone of the text.

The python scripts are responsible for the Deep Learning backend responsible for the core functionality of the project. Text that is inputted by the user is passed through IBM Watson's Tone Analyzer API which conducts sentiment analysis to give the overall tone and emotion of a piece of text. This information is conveyed back to the front end to tell the user of the emotion of the piece of text. Based on the general emotion of the text it is then considered either positive, negative, or neutral and relays this information the data and midi preprocessing scripts which will create the training, validation, and testing datasets as well as additional formatting for the midis. The training, validation, and testing datasets will be used to train, validate and test the double stacked Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) deep learning model that was coded up in the PyTorch Deep Learning framework. The LSTM model has 88 classes and 88 input size to align with the full set of notes in a piano. Optimizations such as adding dropout layers and the Adam Stochastic Gradient Descent algorithm are utilized. Training, validation, and testing of the model was carried out on Google Cloud's Deep Learning services that offered a VM instance for PyTorch and powerful GPU's such the Nvidia Tesla K80. The generated midi format file which contains the song is stored locally and passed along to the Android app, Google Chrome Extension, and Vix website.

Challenges we ran into

With the lack of available midi data sets relating to tone to train the model, we had to manually sift through hundreds of midis and label whether they had a positive, negative, or neutral tone. Since most pre-existing midi datasets have been well structured and formatted, it was difficult to mimic that when creating our own dataset from scratch from various midis. Due to the complexity of the double stacked LSTM network and learning songs, training the model on a laptop was not feasible and required heavy computing power from one of the various Cloud computing services. Due to this, there isn't enough time and resources (cloud computing costs money) to dabble with hyperparameter training and tuning. Learning the ins and outs of the Wix website system was pretty challenging too since it was quite different from website frameworks and developing that we were used to.

Accomplishments that we're proud of

We're proud to have three products: the phone app, the website, chrome extension and pretty functional complex deep learning workflow. With this short period of time, we were surprised at how much we could accomplish and it was fun and challenging to be able to design a system that incorporated so many different things that were able to be connected with one another.

What we learned

Respectively we learned how to make a React-Native app, a Wix Webpage, Google Chrome Plugin, and utilizing Cloud computing services for Deep Learning (IBM Watson, Google Cloud in our case). We all had experience in Python prior to this hackathon but decided to approach new territories and it worked out!

What's next for MusicBoi

Link your Spotify account so MusicBoi can train based off of your type of music. It will then generate music to your taste yet also hold true to the input text. We got the domain but since it was so young, we were unable to transfer it to Wix. When the period is over, the domain will be transferred over to Wix for the webpage.

Share this project: