Netscrape

Web-bottom
Web-top

Inspiration

Web scraping can often be a daunting task, especially to the inexperienced. By providing a way to do this, especially with out inspector-like extension features.

What it does

Netscrape scrapes the net for content within certain selectors (similar to what you would use in CSS/SCSS or jQuery) to look up one or more fields on one or more web pages.

How I built it

We have a Chrome extension, along with a website running a Flask back end and React.js front end.

Challenges I ran into

Some websites generate different content for users, both logged-in and out. Furthermore, some of the class and id names in the HTML are sometimes different based on the users, which means that our servers doing the parsing don't see the same content that our users saw.

Accomplishments that I'm proud of

Being able to automate the process of scraping websites, previously something that could only be done by someone with programming experience, or by paying thousands of dollars for one site.