Inspiration

Being a Full Stack JavaScript developer and already having worked with web scraping technologies in the past. I felt the itch to try and put my experience to use and make Devsite. I came up with the idea when my housemates, who are also developers, expressed an interest making a website portfolio but didn't want to put the effort in to actually create the site themselves. I realized that most developers have a StackOverflow and Github account already setup with their information already populated; so I thought, "hey what not make an easy portfolio generator" and I dived right in.

What it does

Devsite is a webapp that lets you create a portfolio site using only two pieces of information. Your GitHub username and stack-overflow id/username. It uses the GitHub API and scrapes stack overflow's html for relevant information to display. Such as skills, languages, links, profile pic, contributions etc.

How I built it

Language: JavaScript, EJS (templating language), SCSS. Hosting: Digital Ocean [custom Ubuntu server].

ClientSide:

  • Jquery
  • Bootstrap 4
  • FontAwesome
  • GoogleFonts

ServerSide:

  • Node.js 9.00
  • MongoDB nosql database - Mongoose OEM
  • Express routing (for a basic REST api)
  • EJS templating (serverside generated pages)

Development:

  • Gulp.js for task running and an automated build process for SCSS and refreshing of the node server
  • Unit Testing with Mocha.js, Chai.js and Sinon.js

Server Setup

  • DigitalOcean has some amazing documentation for server setup of Nginx [with reverse proxy], LetsEncrypt, Nodejs using pm2, MongoDB and proper authentication, automated VPS deployment using github
  • Domain Hosting: get.tech

Challenges I ran into

I originally wanted to add in LinkedIn as an information source. However I couldn't use a scraper to access the information. I wasn't able to use the scraping technique I was using for stack-overflow to retrieve the information as it seems like the LinkedIn profile page would not load for an automated source like Puppateer (google's headless chromium).

Another issue is I didn't have enough time being a solo coder to complete my automated testing to verify my functions and their full functionality. If I had more time I would have tested everything thoroughly. _ (after the hackathon i'll do just this) _.

Another issue, while I was setting up the server, I made a critical mistake and ended up butchering my user permissions with a misused command. I had to scrap the server and restart. I lost valuable time doing this.

Somethings I wish I added

Web-socket functionality. I wanted to have a server to client socket connection that would have allowed me an easy way to keep the client in the loop with the website build process going on in the back end. I feel that right now the client doesn't know what its going on as all I have is a loading spinner during the build process.

Accomplishments that I'm proud of

The stack-overflow scraping using Node.js's new ES6 async and await functionality was a lot of fun to learn. Being able to pull data from stack-overflow and display it in an organized fashion was very rewarding. Also, having mostly worked with PhantomJS in the past, switching to Puppateer was like night and day.

When I was able to get fellow developers using my site and creating portfolio pages it was amazing to see my hard work come to fruition.

What I learned

During this project I learned a little more about rate limiting from external API's. Working with the newest version of NodeJS and its new fancy tools. Learning more about Jquery html DOM searching. Learning how crazy nests promises can be when not planning out my code

What's next for Devsite

  • I hope to get the LinkedIn scraping system working (code is there), and display job, school and recommendation information on the portfolio.
  • Full automated testing suite with code coverage using Travis CI
  • Update process that weekly checks for new data and updates existing portfolios [currently the pages are mostly static]
  • Full project code documentation (I rushed in the development and didnt have time to document)
Share this project:
×

Updates