We took inspiration from the different categories on Spotify which separates music based on mood. We felt that it might be difficult for a person to exactly pinpoint his or her emotion, so we decided to create an AI to read their facial expressions.

What it does

The website takes a picture of the user's face and passes it into a trained convolutional neural network. Based on the output of the neural network, the Spotify API plays music from the user's account.

How we built it

We built the website using Bootstrap CSS and JS as well as jQuery. We built the Spotify API using the Spotify Java SDK and Spark for Java so that we can have a RESTful server. We built the neural network using Keras and TensorFlow in Python, and we did face detection with OpenCV.

Challenges we ran into

We initially decided to train the neural network on 350 by 350 images, but this was too taxing for the computer's hardware. We decided to switch over to much smaller 48 by 48 images for faster training, but this was a harder dataset to learn since the images were very small. We ran into some issues when developing the camera on the website because of some security features in modern browsers which does not allow the camera script to run on a site that is not SSL secured. We initially tried to implement the Spotify API in JavaScript, but this was a hassle since not many of us were familiar with Node.JS, so we switched over to Java.

Accomplishments that we're proud of

We are proud that we were able to code the entire neural network from scratch and train it ourselves. We are also proud that we were able to make a clean web interface for a very powerful backend. Finally, we are proud of being able to complete the project in the time given.

What we learned

We learned how to save and visualize models in Keras, and we learned how to repeatedly improve our neural network by tweaking parameters to increase accuracy. We learned how to work with video streams in HTML and process them in JavaScript for the pictures. Finally, we learned how to use Maven dependencies in a standalone Java project and host that project as a RESTful service with Spark.

What's next for Music4U

Following this hackathon, we plan to deploy our app on a server so that it is accessible to everyone. We will also add functionality to the app so that users can give feedback about the performance of the AI and thus improve the accuracy of the AI. We also plan to update the UI and to add more features to the UI such as uploading a pre-saved image and a feedback system as already mentioned. We also have a 90% working iOS App prototype that we plan to publish on the app store.

Share this project: