Scrapegoats

• 

• 

  Sign-In page, which is google auth compatible

• 

  Register a new account page, which is google auth compatible

• 

  Profile Screen where you can change your PFP or sign out

• 

  Searching "Apple" by typed query

• 

  Result of "Apple" Search after loading

Inspiration

Public perception of topics through the lens of social media is a heavily studied topic today from both public and private perspectives.

What it does

Our project scrapes tweets containing our given query (received via text or from image recognition via google vision ML) from the past 10 years, and performs sentiment analysis on each tweet before filling a chart with the tweets as data points (x axis is date posted, y axis is sentiment [from -1 (negative) to 1 (positive)]).

How we built it

We retrieved queries using apify's twitter scraper to gather tweets and utilized them on our backend (a flask server which we hosted on google cloud's app engine) where we also performed the sentiment analysis, before using charting libraries (we attempted using both python and javascript libraries) to display the data.

Challenges we ran into

Aside from basic dependency issues, we ran into problems down the line with inadequate database setup that severely limited our ability to communicate between our front and backend. We also ended up running into problems occasionally where a tweet we scraped would contain a language not supported by google's sentiment analysis, resulting in the dataset being unusable for analysis as we did not have functionality built in to clean the data (we tried using twitter advanced search features to only retrieve english tweets but for some reason this did not avoid our issue).

Accomplishments that we're proud of

We are most proud of our concept and its most successful implementation with google vision retrieving the queries to search.

What we learned

We learned the importance of planning the structure of your applications' frontend and backend to achieve efficient, faster, and easier-to-code functionality.

What's next for Scrapegoats

This project could be very interesting for data insight and if we were to look at the project from the start again we could refine how we were using google cloud services (which were very new to us) as well as refine our backend so that we could better bring data between front and backend. This would result in a much more seamless user experience, allowing full enjoyment of our concept of applying machine learning to the analysis of public outlook on various topics.

Built With

• google-cloud-app-engine
• google-cloud-natural-language-ai
• google-cloud-vision-ai
• google-firebase-auth
• google-firebase-storage
• html
• javascript
• python