Mental health illnesses affect approximately 1 in 5 American adults, or about 20% of the U.S. population 18 and older, in a given year. (National Institute of Mental Health “Major Depression”, 2017) We wanted to help mitigate this serious issue through the use of ground breaking technology to detect emotion from speech and video.

What it does

The program asks the user a battery of questions to determine their current emotional state. It sends audio and video data to google servers for emotional analysis, and shows the results in a concise, readable format for a doctor or supervisor to analyze. It also shows its own predictions for the patient's emotional state, ranging from "emotionally stable" to emotionally distressed".

How I built it

We used Python Pyaudio and OpenCV for recording video and audio. We then utilized the google cloud machine learning API to detect presence of key emotions such as happiness, sorrow, and anger. The results are smoothed out and packaged into graphs for doctors to view. The doctor accesses patient files through a portal powered by Node.js and styled with Bootstrap.

Challenges we ran into

One major issue we had while developing the project was getting audio and video from the user. We tried multiple approaches with node-webcam and Node.js, but kept running into errors we couldn't figure out. Eventually, we made the switch to python and were able to get everything up and running.

Accomplishments that we're proud of

Our major accomplishments center around obtaining video and audio from the user. We are able to simultaneously record video and audio, analyze them, and give an aggregated result for emotion over the course of an entire questioning period. Properly weighing and normalizing the data from both text and speech turned out to be a major technological achievement. We are also proud of our integration of google's text to speech algorithms in tandem with their text to emotion AI.

What we learned

We learned how to obtain data from machine learning. We learned a lot about Python and API key usage.

What's next for Depression Detection

Peep the github for updates

Share this project: