-
-
Image of Rate my Dore
Background
About a week ago, I (Ethan Xu) read on LinkedIn that Mike Spivey, CEO of The Spivey Consulting Group and Vanderbilt alum, had released a tool called My Rank. As a law school admissions consultant, Spivey aimed to create an alternative to U.S News and World Report, with a system that allowed the user to rate what was personally important to them in a law school, such as average grant amount, bar passage rates, and diversity. Rather than rely on an arbitrary algorithm's preferences, this method allowed for greater personalization and relevance to each individual user.
With Spring 2025 courses dropping in less than two weeks, my team wanted to apply these same principles to a different dataset: undergraduate professors. While there may a general consensus regarding what a "good" professor entails, there still remains a lack of nuance. Is a good professor professor the one who grades most leniently? The whole who cracks jokes and makes lecture most bearable? The one that doesn't take attendance? Each of these factors could mean everything, nothing, or something in between to any one of the thousands of students at Vanderbilt. With this in mind, we set out create something that could answer these questions.
How We Built
Ethan's Part
The first order of business was to find data. For better or for worse, Rate My Professor was an obvious choice for our project goals. While biased, potentially misleading, and severely lacking quality control, there seemed to be no alternative as far as determine what Vanderbilt students thought about their professors.
Given my background in data science and analytics, I tasked myself with web scraping as my first task. I would began by scraping a Vanderbilt faculty webpage using peak Python, and extracting the names of all the faculty members. Afterwards, I would take this list of names, and run them through a function that would search for the professor on RMP. If found, and if they were not missing data, another function would scrape the RMP page for Name, Department Overall Rating, Difficulty, Would Take Again, the most common tags (Tough Grader, EXTRA CREDIT, Caring, etc), and the number of reviews.
Once all of this data was extracted, I converted it to a JSON file, downloaded it, and proceeded to upload this into a MongoDB database.
Tevin's Part
(Tevin had to go study for a test, so he wasn't able to write this himself.)
Tevin was the sole software developer for this project, and opted to use React and Node.js. To begin, he created a dropdown menu that would allow the user to select which department they wanted to see professors from. After they selected, the user would be presented with a menu with all of the professors associated with that department on RMP. The data for the professors was pulled from the MongoDB database that I set up.
We decided that similar to Spivey's project, we wanted to use sliders and button toggles to allow the user to rate what was important to them. Hence, sliders were implemented that would allow the user to rate how important Overall Rating, Difficulty, and Would Take Again were to them. Their selections would then be passed through a simple algorithm that also factored in how many ratings each prof had, before being presented with a final ranking, from from most fitting to least fitting.
In the end, after having scraped several faculty websites, nearly 200 RMP pages, and compiling everything in MongoDB, we had a working product. Users could perform most of the core features we set out to implement, and were provided with a reasonable ranking that was reflective of their preferences.
Challenges
While I had built some web scraping programs in the past, I never had to parse through HTML. Having never needed to read and understand HTML before, I was stumped, but eventually learned that each page I needed to scrape stored the relevant values similarly, and was able to find a semi-consistent set of parameters that I could adjust depending on the page.
By far the biggest challenge for both of us was working with MongoDB. Neither of us had experience in it, and two hours plus a workshop was apparently not enough to get it set up. In the end, it took completely starting over on the cluster and taking things step by step before we were able to link the database and the program. It was definitely ambitious to try and integrate MongoDB right off the bat, with Tevin telling me at the end that Firebase was much easier and he wasn't sure why we went through all that trouble.
Lessons
This project was my first time working on a serious software project with a team. Coming in, I wasn't sure how I would fit in, given that my teammate was a CS major who had built similar albeit more extensive projects in the past, and I couldn't remember the last time I used a programming language that wasn't Python or R. However things worked out, with the two of us playing to our strengths and trusting the other to do their job well. While we didn't end up integrating all of the features we wanted to, and similarly pivoted away from implementing machine learning, we'd still consider this project a success.
Log in or sign up for Devpost to join the conversation.