Nowadays, information is vast and wide-spread, and being able to visualize data in a graphical format would be a huge asset, considering the following applications: -Business Marketing trends -Primary Educational research resource -Higher Educational purposes -Social Media analytics
What it does
Given a user-inputted keyword, the webapp begins scraping at several image databases, and extends its crawl to as many embedded links as necessary. The result is a list of visual graphics ordered by relevance. Relevance can be influenced by an user-inputted five-star review system. However, Relevance is chiefly determined through the web-crawling algorithm.
How we built it
Challenges we ran into
Setting up the scrappy environment was our first biggest challenge. Establishing a conncection between the front end and backend also became a challenge towards the end of development.The main issue was processing the scraped data produced by the scrappy spider, for display to the front end. Additionally, scalability could also be an issue because all the scraped data is stored locally at this point. Future models will take advantage of amazonweb services.
Accomplishments that we're proud of
Having a spider successfully return scrapped data.
What we learned
How to develop a front end using bootstrap. Project Management (agile). SVC with git.
What's next for Visual Scrape
Debugging, refining the data scraped, establishing an even stronger connection between frontend and backend, adding a functionality to scrape information and present it in infographical format