Inspiration
We love art! Art has been an integral part of the human experience since the earliest cave days where we learned to paint on cave walls. Visual art specifically is both a way to deeper delve into the human condition while simultaneously serving as a time capsule of the passions and musings of a given period. And, art museums are mediums for us in the present to access these freeze-frames of the past!
One way that museums will provide more context to pieces is through audio guided tours. These are fantastic for providing technical, historical background, but we felt that there was something key missing: music! As I walk through a museum, I'd love to be engrossed in the vibes of the painting--to hear and feel what the artist may have been hearing and feeling while creating. So, we created ArtMuse!
What it does
ArtMuse takes in an image of a painting as an input and recommends songs that fit the time period and emotion. For example, if a user took a picture of Starry Night, ArtMuse would recommend dreamy songs from the 1880s!
How we built it
In this v1, we restricted the image set of all paintings that can be identified to 20 paintings from the MET. We used OpenAI's CLIP, a zero-shot image classification model to compare the inputted image to the 20 given paintings and determine which set painting was closest to the input. This allowed up to recover the date of the painting. Then, we used a BLIP sentiment analysis model to pull a "mood" from the painting. Finally, we inputted the mood and the time period of the painting into a script that utilizes the last.fm dataset to find 3 songs that best match the time and mood of the painting.
Challenges we ran into
Both machine learning models that we utilized were new to us, along with the last.fm API. Lots of time was spent finding a dataset that had a usable variety of moods in paintings, picking the best ML model that fit our program, and figuring out the last.fm API.
Accomplishments that we're proud of
I'm very proud of the fact that the program works! This was both of our real ventures into machine learning, and it was awesome to see real applications and relative success of said applications!
What we learned
There is a lot of pre-thought that is required to maximize the efficacy of machine learning models. There are a multitude of factors that affect the correctness-ratio of the algorithm, so learning how to best optimize this program was a super important lesson.
What's next for ArtMuse
Besides deployment side work, I hope to develop this program to collect more data about an image to give more specific song recommendations. This can include data about the country of origin or artistic movement.
Log in or sign up for Devpost to join the conversation.