It was an idea we were all interested in. We enjoy posting photos with fun captions, so why not make a utility that would help us?

What it does

No Cap(tion) is an image comment generator. Users upload images that we then parse and extract what we think the image is about. They then choose the words they feel best represent the image, and these choices are used to generate a list of 5 captions based off of song lyrics that use all or some of these words.

How we built it

The front end of the app is built entirely in Swift. Our backend is built in Python using the Flask framework for serving the API. Image processing was accomplished using the Google Vision API. We then took the data Google returned to us and refined it to prioritize certain terms based off what the primary content of the photo is. These results are then passed off to our song lyric inverted index, which was populated with the Genius API. Each song was ranked based off popularity and relevance, and lyrics that matched the words we're interested in were scored against our algorithm for finding the closest matches.

Challenges we ran into

Correctly formatting song lyrics to ensure consistency. Weighing each artist so that certain ones didn't appear too much or too little. Also, it was our first experience using the Google Vision API and dealing with image processing, so it presented its own challenges. And pulling the best data out of a list of arbitrary data took lots of time to refine.

Accomplishments that we're proud of

Our lyric inverted index boasts a whopping 18,911 different songs. We're also really happy with the accuracy of the image processing and how we can get info that we deem is most important. For example, there may be some times when we want to focus more on parsed text and other times when actual items are more important; getting this right was a fun challenge.

What we learned

Delegation of tasks was key. We planned for a few days before the event and were able to all have a clear idea of what our duties and obligations were, and we didn't step on each other's toes while developing. We knew what we had to do and did it. Also good time management and rejecting scope creep helped us finish on time.

What's next for No Cap(tion)

We'd like to continue to grow the song index. And the processing and evaluating can always be bettered; one can only do so much in 36 hours. For the frontend, it would be cool to incorporate various social media APIs so images and captions can be posted quickly. And lastly, we'd love to actually deploy the app so people can use it.

Share this project: