Inspiration
This is inspired by a personal experience when looking for a therapist. This lead to us wondering how do people normally search for a therapist. We did some research online, and it seemed that most people usually did one of the options: visit their clinic, via their college, via someone they know, via Google search. However, these approaches seemed problematic; for example, by doing a google search it can be overwhelming and unclear because there is no support or guidance in the search. In addition, when searching on Google, the person doesn't really know exactly what is a good or right fit. In most existing websites that list therapists, we discovered that in order for the therapists to be listed, the therapists themselves have to sign up. This means there is a limited scope within each website. We wanted to create a user-friendly and comprehensive website that contains data compiled from our web scraper to provide a larger range of therapists.
What it does
We first created a web-scraper to collect data without having therapists sign up to be shown. After collecting all of the data, we input it into a database. We wanted our website to have a login/profile system so users may log in and bookmark specific therapists they're interested in as well as create a note-to-self section and review history. The website would have guiding questions (i.e. survey) that is a step-by-step guide to help users filter their search. The website will also include an interactive map that shows the available therapists that match their preferences.
How we built it
We used the python library called Scrapy to create a web crawler and scraper. We input the starting url for the crawler to traverse and then the scraper will then save the specific data pulled from the html file into a csv file. We then input the data from the csv file into a database. For the front end, we used Figma to create a mock-up based on our wireframe. After completing the mock-up, we exported the code to create the actual website and hosted it on github.com.
Challenges we ran into
Neither of us had ever created a web scraper, so creating the web crawler/scraper was more difficult than we anticipated. It took us a while to figure out how to extract the specific data from the websites. Some websites had protection against web scrapers, so we had to change our user-agent in order to go around the issue. After we became comfortable with Scrapy, we realized that we could no longer access the website we used as reference - possibly because they mistakenly thought we were trying to send a DOS attack. In addition, we were unable to collect all the data - we believe that is the case because we were timed out; after going through the first few urls, we then got an error that we were unable to access the website, which is why we came to that conclusion.
Accomplishments that we're proud of
We're proud that we accomplished what we our MVP was for this hackathon. In the beginning, we planned out the scope of our project and what our deliverables were, which were: Mock-up design for the website, Implement web crawler, Implement web scraper, Set up database, and Host a simple website.
What we learned
We learned about web crawlers and scrapers, which was a new topic to both of us. We also learned how to use Figma. We were able to go out of our comfort zone during this project and learn new skills.
What's next for STAR
Since we weren't able to create a completely working prototype, the future of this project includes creating the entire website based on our mock-ups in Figma, rendering the data from our database into the website, implementing the remaining features we missed, and scraping data from more websites.
Log in or sign up for Devpost to join the conversation.