TheraBot

Your own digital therapist who listens to your voice, understands your body language and facial expressions to understand you deeper- made for students, here for everyone.

Comment

Inspiration

TheraBot was created with the idea of having a digital therapist capable of offering personalized and empathetic support to address the ever-rising issue of mental health challenges. In specific it was meant for students for a multitude of reasons. The most significant reasons are the rising trends of depression over the years, and the common lack of funds college students typically have. The graph below depicts the trends from 2009 to 2017 with depression rates correlated to age, which have only continued to rise. Depression Rates Graph

Not only do college students tend to lack money, some college students also lack mental health resources. That is why TheraBot provides catered responses based on age and current educational institution to provide nearby resources or possible activities the user can do to help them. All of us also are interested in generative AI, however, we didn’t want to just create another ChatGPT wrapper chatbot. So we used multimodular data for the model to learn from and create even more personalized responses by analyzing the vocal tone, body language, and facial expression of the user in real-time. Overall, we wanted to try out something new for all of us and fix an issue that we see every day.

What it does

TheraBot interacts with users in real time, interpreting their emotional state through different data channels. It was made for students, however could be used by anyone. This is done by capturing the facial expressions and body language of the user through MediaPipe and OpenCV, as well as analyzing the vocal tone using Librosa. All of these inputs are then processed using various frameworks/APIs such as TensorFlow, Keras, and scikit-learn. These inputs are then processed and confidence levels are produced, that information is then fed to LLaMA which then produces responses based upon that information and previous conversation. The session continues in an everlasting loop until the user feels their needs are met.

How we built it

We built TheraBot as a team splitting up different tasks. However, some people finished earlier than others so those people assisted those who were facing issues. Trey Bernard took upon the task of project manager divvying up loads of work and setting timelines that we should approach. He also did all of the front-end work using React, JavaScript, and CSS. Mirza Zuhayr was in charge of the back-end and used Python as well as Flask to deal with the logic and data flow. He also worked with prompt generation using Llama due to how well it fit the task we were taking on. Santiago Roa took on the vocal tone detection using Librosa and TensorFlow together to calculate the needed confidence levels. Brian Pires was in charge of the body and facial language confidence levels which he used MediaPipe and OpenCV to accomplish.

Challenges we ran into

We initially ran into a lack of knowledge surrounding these foreign topics, which is why we spent the first day studying these topics in depth and sketching a plan for the next day. Another issue we faced was a lack of existing datasets, so we ended up having to create our datasets and train our models with this information to get sufficient confidence levels. We also struggled to integrate these models into our backend, however, we eventually figured it out, but it took way longer than it should have. The final issue we had to deal with was fine-tuning Llama to generate the tailored content we wanted, it all worked in the end though.

Accomplishments that we're proud of

We are simply proud that we were able to take on a task that was foreign to us and seemed way too ambitious at first. However, through research and perseverance, we eventually created TheraBot, which we are proud of since it can help those in need ( average college students ). Also, we are proud of the fact we could train a model with multimodular data from differing streams.

What we learned

We all learned so much, primarily regarding data recognition, and how to integrate all of that data to create more effective interactions. Real-time data processing was also a cool topic we all learned a decent amount of, we all underestimated how complex it is. None of us have fine-tuned models with ethical considerations before, so that was also something we learned throughout this process.

What's next for TheraBot

While we are proud of TheraBot, there are improvements to be made. One example can be fine-tuning the model’s sensitivity to understand more emotions and possibly improving the signals. We have a very small dataset compared to what it could be, so creating a larger dataset for each stream of data is a much-needed improvement. We also discussed expanding to mobile simply due to the fact that college students are more likely to download a mobile app rather than visiting a web app. This would be very far in the future, but we thought of collaborations with campus counseling services and their resource directories as well.

Built With

Share this project:

Updates