There is so much to discover in this world. So many great songs, movies, books... However, the content we can identify with the most is what is most relevant to our current stage in life. Content discovery services out there just don't capture this. What if there were a discovery service that would reveal to us new things based on where we are on our personal journey...
What it does
PicktureThis is a photo-based music discovery service. Given a photo, PicktureThis will find a Spotify playlist perfect for the scene. The ideal version of PicktureThis would be a mobile app, where the user can capture a moment they are currently experiencing, and be catered a personalized music experience for that moment. PicktureThis is currently implemented as a web-app as a proof-of-concept.
How we built it
Given a photo, we use Clarita's API to perform object and scene recognition. We then get a list of tags related to the photo. This list of tags is then mapped to 132 music genres using word vector similarity calculations. We used Google's word2vec word vector library to compute similarity between tags and genres. We select the genre that best matches the photo, and find a Spotify playlist in that genre using Spotify's API.
Challenges we ran into
Many Many challenges with getting things set up, especially with installing python packages and setting up the website on Azure. We also had trouble conceptualizing how to find songs given photo tags. It was only after scouring through countless music APIs and realizing the limited search capability they provide did we think of using word vectors. Other challenges were neither of us had much experience programming web apps. There was a lot of learning on the go and troubleshooting, but it was well worth it!
Accomplishments that we're proud of
We are very proud that we were able to get a web-app working. We could imagine that if this app were further developed it would be useful to many people. As a user you can compound on new experiences with new music, and the convenience of photo-based song discovery is definitely very appealing. For example, looking for new yoga music? Just take a picture of your class and a new playlist will be handpicked for you.
What we learned
We learned a lot about web development, getting environments set up, doing API calls, a little bit about word vectors, and developed a lot of patience.
What's next for PicktureThis
PicktureThis would be great as a mobile app. If we can have it interface with photo-sharing apps like Instagram then it would be even more useful to people. Also, music is only the beginning. PicktureThis can extend to catering movies, short films, poetry, art, books... With the rapid advancement of object and scene recognition technology, it will become more and more easy to discover content we can connect deeply with.