With the onset of the Covid-19 pandemic, social media has become a primary method of communication for many students. We often post pictures on Instagram, or Vsco, to update followers on our lives. We often spend hours on end, scrutinizing our pictures and captions and making sure that they will be well-received by our peers. To maximize the amount of good attention that our post may get, we developed InstaFame.
What it does
InstaFame takes in an Instagram post, in picture format, through our website. Our algorithm then detects smaller objects in the image, such as a car or an umbrella, and uses this information to predict how popular your picture may be. It does this by correlating the items in the picture to current topics gaining popularity. The website will return a total popularity score for the image based on the popularities of the individual items over the past few days. In addition to the popularity score, our code also generates several caption ideas for the post, including ones made of inspirational quotes. The captions are currently based on the objects recognized through computer vision, as well as finding terms related to those objects. Social media users could choose different pictures to post until they are satisfied with the predicted popularity score. Additionally, InstaFame prints out multiple captions that the user can choose from, so they can pick the one that most fits their story.
How we built it
We built our algorithm to find a popularity score by identifying smaller objects in the image using opencv, matplotlib, tensorflow, and the Google Trends API. Then, using the most popular item in the picture, we generated captions using wordnets from the Natural Language Toolkit (NLTK) to find quotes for all related words.
We used Django to combine the computer vision and natural language processing algorithms with the website design in HTML. The CV code was written in Python using various libraries and APIs. We used Google Colab and Jupyter Notebook to code and collaborate with each other. As for the presentation and demo video, we used Canva and Zoom, respectively.
Challenges we ran into
When generating captions for the post using inspirational quotes, we had an issue of some keywords associated with the item identified in the picture not having a quote with the word explicitly included. Therefore, we used the Natural Language Toolkit to generate a list of related words for the keyword in the correct context. Using this, we were able to generate more captions with related quotes.
Another challenge we experienced was building the website. None of us had much experience in website development and used tutorials and online resources to learn. We were unsure of how to connect our python algorithm to the frontend aspect of the website. However, we got to work through research and much testing,
Accomplishments that we're proud of
We are proud of integrating the python libraries that recognize various images through computer vision with our html website design. Most of us did not use natural language processing before, so we are proud of adding that component to the project as well.
What we learned
We learned how to use bootstrap and django to create a web application in python. We also further developed our skills to create a visually appealing website using HTML and CSS. Additionally, we learned about many new APIs while coding the python portion of the project. For example, we implemented the Google Trends API, which contributed to the project greatly as it made InstaFame more relevant to current users. This is especially important because social media evolves at a really quick rate. We also learned to use the Natural Language Toolkit to generate related words and alter the output of the lists so we can use it to find related quotes.
What's next for InstaFame
In the future, we plan to make an InstaFame mobile application, compatible with social media applications, such as Instagram or Vsco. We could implement object detection using neural networks, for a broader range of objects that could be recognized. In rating the popularity of a picture, we could also take into account the area that the user is posting from for higher accuracy. Finally, we also want to focus on expanding the caption options generated, to humorous captions or captions entirely made up of emojis.