Suicide is one of the leading causes of death around the world, yet there is an increasing stigma towards discussing how one is feeling. Psychiatrists, psychologists, therapists, and counselors struggle to assist patients because they only see a fraction of their patients' true mentalities, many times a guarded version once a month at a meeting. With the rising cost of health care, more and more patients are not able to attend therapy frequently or do not feel comfortable because of the stigma attached to mental health.

What it does

ThereForYou is an iOS and web app that tracks patients' emotions on a daily basis through speech, tone, and facial expression analysis. After a user simply records their voice or a brief video explaining their day, ThereForYou detects notes of five different emotions: anxiety, disgust, fear, sadness, and joy, presenting the percentage of each detected. Each week, users can see a graph of their emotions from each day. On the web, doctors can log into the portal and access a wide array of information about how their patients are interacting with ThereForYou, how their emotions are doing, and quickly identify patients who are having particularly rough times.

How we built it

The iOS app is built using Swift 3.0, utilizing Apple's Speech Kit to collect a user's inputted words, the IBM Watson API to analyze the various emotional tones in the input, and Microsoft's Cognitive Services Emotional Analysis API to analyze facial expressions and micro-expressions. Combing these inputs, an algorithm reaches a conclusion about the percentage of each emotion. The iOS app calls a backend through Alamofire written in Node.js on AWS to access IBM Watson and Microsoft Cognitive Services and return only the necessary information to the app. The web app is built in Ruby on Rails for a clean interface for doctors to access exactly what they need to know.

Challenges we ran into

The largest challenge we had was getting all of the independent APIs and services to come together into one cohesive, sensical project. We each took on a specific aspect, realizing halfway through that we had no way to connect all of them. Through the backend on AWS, we were able to merge all of the information we had collected and return a result to the app.

Accomplishments that we're proud of

The facial recognition, especially micro-expressions, were incredibly hard to implement, and we wanted a way to make users feel comfortable with what we were asking of them. By including an emoji that reflects how the app sees the person's emotions, users feel more comfortable using the app. Further, the information through the facial expression recognition software is incredibly beneficial for doctors and patients alike. We all feel incredibly proud knowing that we made an app that will positively impact not only our lives, but those around us and widespread throughout the world. Mental health has been neglected for long enough; ThereForYou simply and effectively allows users to track their emotions without stigma and provides that information to doctors.

What we learned

Oh, so much. One of us learned Node.js and built the IBM Watson portion of the backend in a few hours, another spent the entirety of the weekend on the facial expression recognition, getting it to work a mere two hours ago with the help of the whole team, while another one of us focused on connecting the frontend to the backend. We shared our knowledge on various languages, frameworks, and skills. We also met each other for the first time on Friday, which was definitely a plus too!

What's next for ThereForYou

Onto the App Store!

Share this project: