CaptionToLife

• 

  Front page of app

• 

  Generated image caption

Inspiration

We wanted to create an AI-based web app that would allow visually impaired people to understand the content of photographs.

What it does

This project uses an encoder-decoder architecture to caption images, which are then output as audio.

How we built it

We used a ResNet50 model to create image encodings and an LSTM to predict words of the caption from the image and the already partially-generated sequence. We use Flask to create the web app.

Challenges we ran into

• Integrating the image captioning model into a web applications
• Converting the generated caption to audio and have that outputted

Accomplishments that we're proud of

• Successful and functioning web application that is able to caption new images and produce audio with moderate accuracy
• First time competing in a hackathon solo

What we learned

• Flask as a framework for web application development
• The drastic differences in conditioning on partial input sequences alone vs. both the image and input sequence

What's next for CaptionToLife

• Implementing a beam search decoder as opposed to a greedy decoder for better captioning results
• Allowing camera functionality that enables users to take a picture and have it captioned in real-time

Built With

• css
• flask
• html
• keras
• python
• tensorflow