What it does

Put simply, Webscape can make a 3D visualization of the internet by creating a virtual map of linked sites.

How we built it

Using three.js (a WebGL library) and modern HTML5/CSS3 techniques, we were able to render a visualization of site interconnectivity. This information was collected using Beautiful Soup 4, a Python library, which we utilized to scrape anchor tags in the HTML files of a user-provided starting site. Data is requested by the client-side through a WebSocket protocol and is subsequently stored in a MongoDB database located on our Nginx web server (set up through Azure). If the client requests data that's already been requested before, it's pulled from the database in order to save time and computational power.

Challenges we ran into

We each challenged ourselves making Webscape. Steven, who didn't have a lot of prior experience with three.js, encountered his fair share of issues with shaders and node connection. Morgan found WebSocket and Flask to be huge learning curves due to his inexperience in back end, but was also stumped by some front-end troubles. Josh worked primarily on implementation of our collective design thoughts- which must've been pretty hard since we couldn't always agree! Yoseph had the very unique challenge of leading a bunch of rookies while still finding his own problems in building the database and making the Beautiful Soup scraper compatible with Morgan's Flask program.

Accomplishments we're proud of

We each stretched ourselves thin in order to get this project done. Josh and Morgan learned a lot about the cohesiveness of a full stack project. Steven learned more about three.js than he's been able to in his own time this semester and that was a goal of his. Yoseph was incredibly proud of his Python web crawler.

What's next for Webscape

There are improvements that could be made to the UX of this project. We'd also like to add functionality for metadata crawling- when the user clicks on a node then site data is displayed in the side panel. We have a lot of ideas, and we'd love to talk with you all about them!

Share this project:
×

Updates