Text Analysis (NLP)
Vision Analysis (CV)
Vision Analysis Graphs
Audio Analysis Spectrogram
Segmented Playback (2)
According to the World Health Organization (WHO), 1 in every 5 college students suffer from mental health disorders including depression and anxiety. This epidemic has far-reaching personal and societal consequences – increasing disease risk, fracturing relationships, and reducing workforce productivity.
Current methods of depression diagnosis are extremely time-consuming, relying heavily on one-on-one clinical interviews conducted manually by psychiatrists. Moreover, the subjective nature of evaluations often lead to inconsistent medical advice and recommended treatment paths across different specialists. As a result, there is a clear need for a solution for efficient and standardized depression diagnosis.
What it does
SmartPsych is a streamlined web platform for automated depression diagnosis from videos of natural patient conversation. Leveraging a unique mix of audio, computer vision, and natural language processing deep learning algorithms, SmartPsych accurately pinpoints and quantifies the complex network of symptoms underlying depression for further psychiatric evaluation. Furthermore, SmartPsych has a special sentence-by-sentence playback feature that enables psychiatrists to hone in on specific sentences (color-coded for severity from green to red being the most severe). This enables psychiatrists to identify the patient-specific struggles and affected topics – paving the way for personalized treatment.
SmartPsych is currently able to detect depression sentiment from patient words, negative emotions (e.g. anger, contempt, fear, sadness) from facial images, and valence and arousal (i.e. positive energy levels) from spectral qualities of a user's voice.
Overall, SmartPsych represents an entirely new way of depression diagnosis – instead of wasting the valuable time of psychiatrists, we delegate the tedious task of identifying depression symptoms to machines and bring in psychiatrists at the end for the final diagnosis and treatment when their expertise is most crucial. This hybrid man and machine model increases efficiency and retains accuracy compared to relying on machine learning predictions alone.
How I built it
The web application was built entirely in Flask; to facilitate user interaction with the application, I used the pre-designed UIKit components to create a minimalistic yet user-friendly front-end.
Since current sentiment analysis methods are unable to directly infer depression, I trained my own CNN-LSTM neural network on the 189 interview transcripts of both depressed and non-depressed patients from the USC Distress Analysis Wizard-of-Oz database. After undergoing supervised learning, my neural network had a validation accuracy of 69%, which is highly comparable to current human inter-rater variabilities between 70-75%.
Emotion classification and facial landmark detection was performed through the Microsoft Cognitive Face API, valence and arousal prediction was performed through multi-variate regression on audio Mel Frequency Cepstral Coefficients (MFCC), and speech-to-text transcription was performed through Google Cloud.
Accomplishments that I'm proud of
Integrating custom-trained Keras/Tensorflow models, Google and Microsoft cloud APIs, and regression trees into a single unified and streamlined application.
What I learned
The sheer amount of factors behind depression, and calling cloud APIs!
What's next for SmartPsych
While SmartPsych can now be easily used on the web for effective depression diagnosis, there is still much room left for improvement. Development of a mobile form of SmartPsych to increase usability, integration of other biometrics (i.e. sleep quality, heart rate, exercise) into depression diagnosis, and increasing understanding of depression sentiment prediction with entity analysis.