There are 2 challenges we face when we want to upload an image to Social Network (Instagram, Facebook etc.,)

  1. Image Filters
  2. Caption for the image

There are 1000s of applications that add filters. But, not a single one which provides you with RELEVANT captions.

What it does

We have developed an application that briefly does the following - 1. User uploads an image from the app 2. Server processes the image - Data processing pipeline 3. User gets to see the most relevant Quote or Story of the picture as he opts. Also, as an addon he gets a list of relevant hashtags.

How we built it

We developed the app using - React Native Server Technologies - Node.JS, Python Machine Learning/Deep Learning - Tensorflow, Amazon Rekognition, CNN, RNN AWS stack - EC2, S3, RDS, Elastic Search, DynamoDB, Route 53

Here is the workflow- 1. User uploads an image from the app 2. The image goes through a Data Processing Pipeline a. Label Detection - Detect what objects are part of the image. b. Context Derivation - Labels don't signify anything! They are just random words. We perform multi-level correlation(Recurrent Neural Networks) and gets the context that defines the image. ex- For a given image Labels- bike, sun, road, trees, helmet, shoes, person - they don't make sense. But, what we get out of context is - 'A bike trip' or 'A long ride' c. Hash tags Derivation - This is purely on the label, we get the most 'trending' hashtags for the given label d. Quote/Story - We use Elastic Search and the power of GPU to derive the most relevant Caption/Story for the picture. e. User finally shares his story across social networks.

Challenges we ran into

  • We had issues while we were building the correlation engine. We needed huge corpus of data to train our models. We had to build the dataset by ourselves. This was not just time consuming, but, tiring as well. But, we did build tools around it to automate the processes eventually. Now, we are all good. The accuracy is also pretty high, near perfection.

Accomplishments that we're proud of

  • Some of the TOP celebrities use our application. Interesting part was, the celebrities were really interested in the app.
  • We have a conversion rate of 84%(last week). 84% of the people who upload an image to get a caption, have shared it on social media.
  • Graduated from YC.

What we learned

What's next for AutoCaptions

We had also applied for YC. We have graduated out of YC. This is a proud moment for us.

Share this project: