In an increasingly digital age, many of our most treasured moments--weddings, honeymoons, birthdays, graduations-- are captured in digital photographs. However, large albums can be difficult to compile and edit, especially in a way that truly encapsulates the moment. Fortunately, music has an immense ability to stir up powerful memories and emotions. Our team wanted to create a photo-editing app based on this power. In doing so, we hope to make more cohesive, powerful photo collections.
What it does
FineTuned is a web app that allows the user to both upload images and search for a song of their choice. The app edits each photo to match the mood of the song, creating a cohesive album of photos.
How we built it
FineTuned leverages the Spotify API to fetch song data based on a query composed of the track and artist. We also utilized the Genius and Microsoft Azure's TextAnalytics APIs to fetch and measure the sentiment of each piece of music. These pieces of data feed into various tuning algorithms that edit the photos as necessary, using PIL. These functions are connected through a Flask backend. Through Flask, the user fills out two forms, one for photos and one for the song they wish to select, before the user is redirected to a page where they can view their edited photos.
Challenges we ran into
One of the challenges we had to overcome when creating FineTuned was how to use the data we got from the Spotify API to edit the pictures in a way that reflects the general tone of the song. There are many moving parts that come together to create the tone of a song, a low energy song is not necessarily a sad song while not every high energy song is happy. This created a huge problem when we tried to edit the images as sometimes the edits would not fit in with the general tone of the music. We challenged ourselves to write an algorithm that would do the song justice and have noticeable differences between songs even if they were both either happy or sad.
Accomplishments that we're proud of
Our greatest accomplishment was overcoming our biggest challenge, finding a correlation between the image edit values and Spotify values. The key was using multiple track values to determine the perfect saturation, brightness etc. Another API that helped us find the general mood of the song was the Genius’ API. The Genius API helped us get the lyric sentiments of the song which aided in helping us decide the direction to take the editing in.
What we learned
For some members of the team, this was a first dive into Flask, Jinja, and APIs, all of which were crucial to the project. Due to these differences in experience, we also learned to collaborate, filling in for each others' weaknesses so that the end objective could be accomplished.
What's next for finetuned
Some immediate next steps we could take involve making a more user-friendly interface for users to download their edited photos, as well as further working through the details of the tuning algorithm.
Long term, AI could have play a potential role in the photo-editing process. Currently, the coefficients used in photo-editing are hand-picked; however, with more sample data, it is possible that neural networks could make the process more consistent.