What it does
spotif.ai determines the best song for a listener based on their facial expression. The listener can choose to either stabilize or boost their mood, and our app determines the most appropriate song using metadata from a data-set of song attributes from the top 2000 songs on Spotify in 2017, and links to that song on YouTube. It selects a song based their happiness level as calculated by Google Vision's face recognition API.
How we built it
Our application was built almost entirely with Android Studio. Here we built the UI and back-end interface of it. We also integrated Firebase APIs(Realtime Databases and ML Kit). The database served to hold data from our song data-set so we can easily reference it in the app. The ML Kit allowed us to make inferences about a user's emotion based about their smile level.
Challenges we ran into
Our first major hurdle was figuring out what data was needed to pick the most appropriate song. We had very few useful inputs we could pair with the app. Research was key in order to get past this hurdle. Another major problem was finding a way to pair heart rate data with the emotions detected by a camera and use both of them to determine the final song valence. Pairing the data was intensely difficult because the only sensor we had was a Fitbit Ionic watch. The APIs for it was unfortunately incompatible with Android apps. After hours of trying a lot of adjacent possibilities(talking to mentors, running a web engine, creating a node.js server with google cloud functions), we compromised and decided that we did not have enough time to figure out how to get data from our watch to interact with data from the mobile app.
What's next for spotif.ai
spotif.ai has a lot of things going for it. Not only can it revolutionize the way we pick songs and listen for fun, spotif.ai can also have applications in psychological therapy and educational settings. In the plan to develop machine learning models that is specifically tailored to users. Along with this, we definitely plan to increase our database of songs and further improve upon the UI by connecting with more music platforms(SoundCloud, Pandora, etc). In the future, we might even be able to integrate the heart rate sensor and other inputs which can further improve our recommendation system.