CrisisDetector

Dataset of a total of 18000 rows
Word Frequency Cloud
Time of Posts
Survey Data
Sentence Similarity Scores

Inspiration

Over 90% of college students in the US have experienced some level of negative mental health symptoms according to a BestColleges.com survey. This has been aggravated by COVID-19 and the advent of online education, increasing mental health concerns in the student fraternity. Often, students have a hard time expressing their mental state to their family and friends, and thus their condition goes undiagnosed which can lead to concerning outcomes.

What it does

With this in mind, we wanted to create a tool that can detect concerning trends in social media posts and take appropriate action if the concern levels are high. We believe implementing these detectors on different social media platforms can act as a proactive tool for detecting early signs of mental illnesses.

How we built it

We wrote a python code using beautiful-soup and tweepy to scrape Twitter and Mental Health Forum for posts. Using those posts, we created a bag of words that are frequent in posts that show clear indications of mental health issues. Then, using the cosine similarity index we wrote a code to detect similarity in new posts to look for indications of severe mental health issues that could show suicidal tendencies.

Challenges we ran into

With no experience in web scrapping, we were looking for teammates with some experience in web scrapping. However, we were not able to find anyone who is willing to join our team. Hence,... Alt text

We also tried crawling, but the code was taking much time to run—with no errors ;)—which we thought is not suitable for the scope of our project.

Okay, well, websites do not like being scrapped—I don’t think anyone would—which was difficult to get around because often we were required to verify that we’re not bots.

One important feature we tried to add to our tool is the geo-location of the tweet or post, which will help in sending the OP’s information to schools and colleges in that area to contact them. This will allow schools and universities to take proactive action in aiding students with mental health issues. It is very common that students rather post about their issues online than seeking help with their counseling centers.

We tried to create a flask server to link our code to HTML, but, unfortunately, time was of the essence and we could not do it.

Accomplishments that we're proud of

We are very proud of the work we were able to accomplish especially in the area of web scraping. We, literally, learned web scraping in a few hours. None of us had any previous experience in web scrapping—our expertise is limited to data analysis. We were able to generate ORIGINAL, UNIQUE data sets via scrapping Twitter and Mental Health Forum. We were able to scrape over 300 pages from the Mental Health Forum, which was time-consuming (40 mins!), with around 8,000 rows of data about posts. From Twitter, we were able to generate around 10,000 rows of data that is updated every time we run the code with real-time data.

What we learned

We learned about web scraping, its limitations, usefulness, and verification. It is vital to have a human element in verifying the validity of natural language processing models because machine learning often cannot grasp the context of posts and might produce false positives or false negatives.

What's next for CrisisDetectors

One important feature that could further enhance our tool is incorporating geolocation detection of posts. This will allow us to detect the location of the posts, and we can send this kind of information to schools and universities in the area. Schools should start searching for those usernames in their databases and reach out to help them—being proactive.