O.L.I.V.E.R.

Revolutionizing mental health therapy

Comment

Inspiration

From having personal experiences with facing depression and knowing numerous people really close to us who had to fight this battle, mental health has always been a serious topic that we have wanted to revolutionize. We realized that people around the world, including students, are facing more stress every day and are having a difficult time sharing their experiences with the people close to them. Approximately 16.2 million adults in the United States are diagnosed with depression and suicide is the second leading cause of death for young people aged 15 to 34. This topic, being really close to our hearts, is important and drove us to try our best in revolutionizing the way people deal with mental health problems such as depression.

What it does

O.L.I.V.E.R. is a virtual therapist that simulates the responses that a professional mental health specialist would respond with. O.L.I.V.E.R. monitors the user's emotion based on the use of a webcam library and custom trained Deep Learning classification model. By utilizing a photo-realistic environment in Unity, we are able to accurately simulate a real face-to-face therapeutical session. After doing some research, we found out that face-to-face conversations activate specific neurotransmitters in a person's brain that enhance their experiences. We use the power of Natural Language Understanding with Google Dialogflow to provide an authentic conversation.

How we built it

We used Tensorflow, a deep learning framework, to learn a model that would map the image of a person's face to the emotion they are feeling. To achieve this within the time constraints of the hackathon, we performed transfer learning on an Inception ResNet classifier trained on the ImageNet dataset. To do this, we loaded the pre-trained weights for image net, then replaced the classification dense layers with our own custom model that we trained using data obtained from Kaggle.

We created an assistant on Dialogflow to provide an authentic conversation to the user, and we spent extensive time and research to formulate effective responses and trigger phrases for the system.

The Unity simulation allows the user to interact with the rest of our system in a realistic manner. We utilized photo-realistic models to make it seem as real as possible and to make the user feel comfortable as well. We integrated the Unity game with Dialogflow by using the Google Dialogflow C# API and connecting it into Unity. The audio is streamed through Unity into Dialogflow, and the result is streamed from Dialogflow through Unity back to the user.

Challenges we ran into

We faced some difficulties integrating Dialogflow with the rest of our system. In addition, our deep learning specialist ran into troubles with overfitting the dataset due to limited data. We were unable to integrate our deep learning model with the rest of the system due to a lack of a cloud server to deploy our APIs to. Instead, we built a system API that would be able to be ported to TensorFlow serving which would allow us to deploy our model to production.

Accomplishments that we're proud of

We are proud of how realistic the Unity simulation turned out given the time constraint. We were also proud of how we were able to train such a large Deep Learning model with the lack of computing resources and time. In addition, the Dialogflow portion of the system worked out effectively.

What we learned

We learned how to correctly delegate tasks between ourselves based on the skills we each had and also based on what each of us wanted to learn through the course of the hackathon. We learned about how to integrate Dialogflow and other internet APIs in Unity through C#, and learned about the struggles of training a Deep Learning Network effectively on low amounts of data, and how to solve them.

In particular, we learned about many regularization techniques that deep learning practitioners use to improve their model's accuracy on the test data. Some of these techniques include L2 Regularization and Dropout, and we also learned about the art of transfer learning.

What's next for O.L.I.V.E.R.

Development wise, we plan to add Augmented or Virtual Reality components to our simulation to increase the authenticity of our product, and also increase the appeal to our users. We also plan to incorporate Reinforcement Learning techniques to our system to have our system constantly learn from its experiences with its patients. This agent will learn from the user's vitals and emotions to understand the effect its actions had on the patient's mood.

As for the marketing and deploying side to this project, we plan to start off the product by licensing it to medical companies as an initial information gatherer from patients which would streamline therapy sessions. By licensing to companies, we surpass the process of marketing our product to consumers and bypass building a consumer base from scratch. Once we have some data and more time to make our system more complex, we to increase the scope of the system by making it self learning and then to be deployed as a full end to end therapy system, and to substantially decrease depression and other mental illnesses.

Built With

Share this project:

Updates