Inspiration
One of our members plays Magic the Gathering and would often purchase cards from an online card shop. He has always found it tedious that he has to compare prices from different websites to find the most affordable price. So he got the idea to create a web scraping project that can easily aggregate data from all of these shopping websites to one.
What it does
The web site scrapes data from online trading card shopping sites (currently we're scraping from three: KanataCG, 401Games, and FaceToFaceGames) and aggregates them to one site so that we can easily compare prices from multiple sites.
How we built it
For the front-end we used the ReactJS framework. For the back-end we mostly used C++ for the data processing with a little Python for the web scraper. We used Redis as a cache database to temporarily store data.
Challenges we ran into
Since we're relatively new to working on GitHub as a team, we had trouble with all the branching and merging. It severely stunted the project development as failing to merge the some of the pushes made it so that the front-end part couldn't be properly tested.
Accomplishments that we're proud of
We're proud that we're manage to produce a somewhat functional project despite using many tools for the first time such as Redis.
What we learned
We learnt several new skills such as Redis and web scraping. We also learned that we need to study more about GitHub branch management.
What's next for TCG Card Web Scraper
If we let the website run for some time we'll get more and more data about card prices. We're hoping that we could eventually add a machine learning component to perform data analytics on the card data such as seeing price trends and predicting price fluctuations over time.
Log in or sign up for Devpost to join the conversation.