It is said that a picture is worth 1000 words. It is well known that some feelings and emotions can't easily be captured by prose. But what about music? Nostalgia, joy, sorrow, heartbreak; all these emotions have been expressed eloquently by the musical geniuses of our time. We were inspired by the Spotify machine learning session and Google cloud's Vision AI to try to combine music with pictures to allow a user to immortalize memories in a unique way. What if we could listen to music and feel the feelings associated with a memory?
What it does
Spotifind turns an image into a playlist based on emotion and key objects in an easy four step process. Step 1: Login to your spotify account Step 2: upload your favorite image (or any image at all) Step 3: De-select any objects that you don't want to influence your playlist Step 4: Recieve a playlist of 20 songs to enjoy! this playlist is also directly added to your spotify account's music!
How we built it
The backend of Spotifind was built using Flask and the front end webapp was built using ReactJS. Google Cloud Vision API takes in the image in the given path and produces texts that signify labels (or objects), emotions, and colours detected from the image. If there are no people in the image, it analyzes color instead. These keywords are sent to the frontend, and presented to the users so that they choose keywords that they will base their playlist on. Then the keywords and the user authentication are sent back to the Spotify API, and after the user's account is authenticated, they are weighted and used to (i) search songs whose title is related to the keyword, or (ii) take recommendations from Spotify's recommended playlist. Then Spotify API is again used to create a playlist of songs found and return the url of the playlist to the frontend.
Challenges we ran into
Realizing that we need a separate authorization token on top of API access token was one of the most challenging parts. We had Keyword Error bugs as we started calling APIs that required access to a specific user's Spotify account, instead of searching through a public playlist or a database. We were able to fix the issue once we realized it, along with other minor syntax errors.
Accomplishments that we're proud of
That we got it a minimum viable product done. This was definitely an ambitious hunk of bread that we bit off and we learned a lot along the way.
What we learned
None of us had used Vision AI before and we were very new to React as well.
What's next for Spotifind
Spotifind has tons of room to grow. The Spotifind song recommendation algorithm could improve vastly with additional integration of the user's preferences. Using the key terms to suggest songs based on song lyrics as well is another addition that we could make. We would add having the image set as the playlist cover. Playlists based on video could be added as well; videos could be scanned for key frames and then those frames could be processed and their attributes weighted and aggregated to make a playlist. Additionally, we conceptualize potential partnerships such as Spotifind partnering with BeReal so that every time you take your daily picture you get a daily playlist to listen to on the go.