A great way to improve mental health while in isolation is by learning, and with so many online courses being offered for free during the COVID-19 pandemic, now is the best time to learn at home. CourseHub makes it easy to do this by listing free courses from a variety of hosts all in one place.
What it does
Course Hub searches multiple hosts on the web and finds courses being offered for free. The courses are presented in an easy-to-read list including links to each one.
How I built it
We use Selenium, a Python web-scraping library to get course information from the web, and store it in a MongoDB Atlas collection. Since the application is Python-based, we chose to use Flask as the platform for our web page. The Flask application reads and display data from the entire collection. We also used Bootstrap to make the web page more presentable.
Challenges I ran into
Each host that we retrieved courses from obviously have different web page structure, meaning we needed a different web scraping function for each one. Our team had very little experience using MongoDB, which made it difficult to implement the search function.
Accomplishments that I'm proud of
We were able to tackle some technology that we weren't familiar with and managed to build exactly what we imagined from the beginning.
What I learned
How to use Python for web scraping, and how to use MongoDB as a back end database.
What's next for Course Hub
Implementing a way to filter by host or by subject would be useful for finding courses quicker. The web scraping function could perhaps be improved by using multithreads. Currently, it takes about a minute and a half to execute the function and write to the database. There's many more hosts offering free online courses that we didn't include in the project. We could update the project in the future to scrape from more sites and list even more courses.