Our inspiration for this project stemmed from our desire to build a cool web application using Machine Learning and Cloud technologies. However, we were wise enough to not aim for the moon straight away and have nothing to show for it so we decided on an iterative process where we start with a core identity that we can add features to as we progress. Therefore, we settled on the humble idea of starting with classifying the mood of a song that is retrieved from the Spotify API and then use the Bing Image Search API on Azure to serve a collection of images that are related. Eventually, we were able to add more features that will be explained in the next sections.
What it does
As mentioned above, Moodify start by connecting the user to their Spotify account and allow them to request a song. The song is then passed to a Tensorflow machine learning model that will predict the mood of the piece between Happy, Sad, and Energetic. Then, we also utilize the lyricsgenius python library to pull the lyrics of the song that we process, before sending them off to IBM Cloud's Natural Language Understanding service that returns the keywords of the song. Using the mood and the keywords, we construct a few search terms that we send to the Bing Image Search API which in turn gives back URLs to relevant pictures. Finally, we serve the audio of the song as well as the images simultaneously on the front-end for the user to enjoy.
How we built it
The backend we used is Flask and we used the Tensorflow library to load the model. We also connected our application to a variety of APIs such as the Spotify API, Bing Image Search API, Genius API (through lyricsgenius), and IBM NLU API. Furthermore we used SQLite3 to store our user data and SocketIO to continuously serve images from the backend to the front-end.
Challenges we ran into
- Settting up the project.
- Spotify Player
- Showing/stop showing the images (SocketIO)
- API tokens expiring / limits getting hit
Accomplishments that we are proud of
We got the core functionality completed.
What we learned
What's next for Moodify Player