SmartScrapes

SmartScrapes is a web application built using Flask which provides an analytical solution for the companies who want to analyze their product.

About The Project

While building products we often require feedback to improve but to take in 100 reviews at a single time is pretty tough to remember. Artificial Intelligence has transformed a lot of industries in the past and hence it makes sense to transform this arena as well. With SmartScrapes, these reviews are processed quickly using the power of Natural Language Processing which gives us a quick, efficient bird's eye view of the performance of the product involved.

How it works

  1. The user can enter their Twitter handle and the NLP model then analyzes it for them 15

  2. Next, if the user has their app on PlayStore or the AppStore they can enter those details as well. 16

  3. The users can also read reports which is a more textual overview of the reviews. 17

  4. If required, they can upload their own data 18

Examples

This project is deployed at https://smartscrapes.herokuapp.com/ so do check it out.

Challenges involved.

  • Making the app was definitely not easy. It is easier to analyze the data that at least won't change too much but with this application I did knew the fact that different products would be involved and hence it took time to select the right set of visualizations to show.
  • After that, I would say the next difficult part was integrating these visualizations with the web app itself. I wanted to make sure my visualizations are interactive and possible to save on the device and hence I turned to Plotly but integrating it was really tough
  • Finally, I would say since this app heavily relies on NLP. I had to make sure that the response times were fast. Initially, it would take around 2-3 minutes for making inferences on the reviews but after optimizing a lot of things in my preprocessing steps and the main inference involved I was finally able to make it usable and deployable.

Next features.

  • For this application, my next plan is to be able to compare two products together at the same time which would be even more beneficial for the users.
  • I am also planning to work out the way for converting the reports generated to an automatic pdf version which would take away the hassle of preparing product reports.
  • Final thing I am planning to work on is for the app to be fast on more larger datasets in the custom data section.

List of usernames to try out!

  • Tesla: A company founded by Elon Musk. Would be cool to analyze the online presence of a big company eh?
  • PlayChoices: A mobile app that lets you play stories. Pretty cool app and I do play it:)

Tips while viewing the visualizations on the app!

  • Make sure to have at least a basic Twitter account for your brand to analyze it.
  • If the visualization doesn't load up soon try it once again. It might be because of pre-processing delays
  • If the visualizations seem small try toggling the zoom values. They are responsive so it would definitely come into the correct shape :p
  • Finally, it is not just a static visualization so play around with it and save it up on your device afterward!
Share this project:

Updates