We wanted to use one of the many APIs (Spotify, Twitter) to build a machine learning algorithm that would effectively determine personality or sentiment. We also wanted to create an immersive, artistic experience that could take this AI to the next level. We ended up going with Spotify because we felt that music taste was the most personal, intimate reflection of a person, and built CAI, a machine learning algorithm to classify you in this vast city of Termina.
What it does
CAI will ask you to log in using Spotify and will gather your recent listening history and saved tracks. Based on the Spotify audio features from each track (liveness, valence, energy, etc.), it will classify you within one of twelve character archetypes typically found in the cyberpunk genre.
How we built it
We went over a number of methods to determine personality, reading over extensive research material and learning about various personality measurements, like the Big Five or MBTI and how it may correlate to Spotify music. We didn't find the answers we were looking for, so we finally decided to build our own set of determinants from scratch.
In order to run the initial clustering algorithm to determine audio characteristics of the twelve personality types, we used this Spotify dataset of 160k songs both past and present. Next, we ran a simple k-means clustering algorithm in scikit-learn to label each song in the dataset with a label 0-11.
Using this new dataset, we trained a simple Random Forest Classifier model from scikit-learn using a random sample, and tested it using a different sample from the same dataset, achieving an accuracy of 98%. Then, we retrained the model on the entire dataset and serialized the model, so that the full model can be used on new data from the user's Spotify listening history through the Flask API.
The frontend is built in React. All the art found in the web app was completely made by us, save for the Spotify album covers and Spotify codes.
Challenges we ran into
It was definitely difficult coming up with the original algorithm, as personality detection from user account data had not been done like this before. We also needed a way to match personality up with a set of audio data, and this took a lot of time to figure out.
The API (both Flask and Spotify Web API) proved finicky. This was the first time we worked with the Spotify Web API, and the user authentication flow was particularly error-prone at the beginning. However, we learned a lot about APIs, requests, and more in Flask and Python.
We also made the decision to handdraw each character. Since we had decided on twelve options, we needed to sketch and color twelve characters in 24 hours.
Accomplishments that we're proud of
Each character's description and image was created by ourselves. Considering the amount of detail is in each character, we are very proud of what we were able to accomplish in just a day! Second, building a machine learning model for personality detection from scratch and pipelining it to an API for production was also something we had never done before. Some of us never worked with React before as well, so building a full web app in React was challenging but to be able to do it was invigorating.
What we learned
Building APIs in Python using Flask, using React, Spotify Web API including Spotify Authentication Flow, animation, cyberpunk theory, uses of various personality measurements, producing machine learning models for production, various clustering algorithms.
What's next for CAI and Termina
We're excited to improve the full user experience. This includes
- Adding music previews as the user's library comes on screen
- Implementing different chat functions using the Google Text to Speech API
- Bold animations
- Refining Termina's narrative and story
- Better error handling
- More security functions
- Better use of React hooks
In addition, we want to move our program to better-suited production servers, instead of hosting on Repl.it (which is not a convenient place to host for production). We also hope to iterate on the existing algorithm, so it's more accurate and a better reflection of a person's (or android's) true cyberpunk soul. We see the method we come up with expanded to other places—sentiment and emotion analysis using the Twitter API, restaurant or food taste, etc.
CAI has ambitions beyond Termina. CAI hopes to rule the world. (We're kidding. We hope.)