SLFYcare

Enter SLFYcare, the smart blog with sentiment detection where you can upload daily diaries with selfies to detect sentiments with facial recognition and NLP!

Comment

Inspiration

Last week, one of our team members experienced a close friend pass away due to a sudden illness. He told us, "even though I did not try to show my grief to others, it was apparently visible to my parents that I was struggling with depression from losing a loved one in my life. Had I not opened up to them, I am not sure how badly the grief would have impacted me."

We wanted to use facial recognition software and NLP techniques to create a self-care blog that can pick up nuanced cues in a person's expression through both through facial expressions and written words. This information can be used to detect underlying emotions in a person and encourage them to seek help and reach out to others when they are in need of mental help.

What it does

Meet SLFYcare!

This smart blog:

…lets you write daily journal entries to help you recognize thought patterns and common themes and analyzes those entries to find common themes that you might not be aware of. This knowledge can lead to a deeper understanding of your emotional state.

…takes daily selfie entries to help you recognize your emotions through visualization and classifies emotions in your expression that you might otherwise struggle to name. Putting words to your feelings can help identify the true causes behind your mood.

… combines the data of both your journal and selfie entries to give you a comprehensive overview of your emotions over time so that you can monitor your progress and stay on track on your way to happiness.

demo link here: https://streamable.com/0exdx

How we built it

Catherine: I used a pre-trained Keras convolutional neural network to identify emotions from images and extracted heart rate variability (HRV), workout duration and nightly hours of sleep from a team member's Apple Watch and analyzed in Python to track trends over time.

Vincent: I incorporated NLP-based clustering algorithm provided by Textacy, a SpaCy-based Python library developed by chartbeat-labs to identify keywords found in journal entries and included the text-classification function presented in SpaCy to compute emotive scores for each journal entry. For the text classification algorithm, a mock data was generated from a sample database of IMDB movie comments used by SpaCy to train the model on spot, then the model was used in predicting user emotions. Along with Sunny I worked on full-stack development of the project to ensure that the frontend, responsible for capturing selfies and submitting journal entries, communicated smoothly with the backend, where the three machine learning algorithms ran on accumulated user data.

Kathrin: I contributed very little to the building itself. The only thing I helped with was gather some visual data and create text entries that my teammates used to test the model.

Sebin: I first started off with creating a persona and thought about the problem that our audience is having a problem with. To solve the problem, I started to sketch out the possible interface outlines. After the outline, I came up with the brand identity and logos. Then I started to integrate the brand identity to the interface that I sketched out. After the design part, Sunny and I start to work on the implement the design to frontend development. After then, I jumped into build a deck for the presentation.

Challenges we ran into

Vincent: Lack of frontend development experience led to many hurdles while trying to host SLFYcare. At one point I was stuck for 5 hours trying to solve a CORS error and thought I was doing something wrong but it turned out the error showed up because I forgot to turn on a certain plugin on my browser.

Catherine: We also ran into issues with our emotion recognition algorithm - though it ran, it tended to confuse similar facial expressions, like disgust and anger, and had trouble detecting faces when the user was wearing glasses.

Kathrin: My biggest challenge was figuring out how I could help my team with no knowledge in developing or coding.

Sebin: We ran into a challenge where we try to implement the camera into the interface that we already designed. Since we did not develop the functionality of the camera yet, we had to think backward to figure out where would be the best place to insert.

Sunny: We ran into challenges of integrating web cameras to capture correct frame for image recognition. I tried using aai and raspberry pi but recording the frame on flask was challenging.

Accomplishments we're proud of

Vincent: With great challenges come many learning opportunities. The lack of frontend development experience also meant a great chance for a learning experience in terms of building and hosting a frontend server and enabling it to communicate smoothly with the backend server, which ran the two machine learning algorithms (Facial emotion detection and NLP-clustering) and sent results regularly to the frontend. We also efficiently used task division among our members to work efficiently and collaboratively.

Catherine: I'm quite proud of doing my entire analysis in Python - most of my analysis experience is in R, and I've been looking for an excuse to do something entirely in Python, so this was an amazing opportunity!

Sebin: Definitely our role division. Because of the fact that we met each other at HackSC, we had to face and learn what would we as a team are capable of and doable within a short amount of time. No one was lazy or laid back, everyone worked hard to finalize the project.

Sunny: I think I am very proud of the team, we stuck together, understood each other and most importantly had the most fun. The temperament of every team member was amazing and this was one of the main reasons I came to this event. We are proud of what we have built.

What we learned

Vincent: Role division is crucial! Had we not started each day with a clear cut role division we would not have completed this task. Though our roles changed throughout Saturday as we juggled through different tasks, we were able to tackle each hurdle one by one with good role division with manageable workload.

Catherine: I learned more about convolutional neural networks, both in terms of having a better understanding of what they are and how to implement them.

Kathrin: I got a much deeper understanding of what I still need to learn (coding languages, UX/UI programs, how all of it works together, etc.) That might sound like a very small learning but it meant a lot to me that my teammates took the time to help me learn and find some starting points into this area that I find so fascinating.

Sebin: Teamwork was the greatest outcome amongst all. We learned how we communicate and ask for needs and helps. Even though there were some hardships involved consistently, we problem solved task by task.

Sunny: We learned teamwork. Coming from various backgrounds with different skills and working towards a common goal that we all wanted to work on was an amazing experience. Technically I have improved upon my troubleshooting skills and getting things done in the knack of time.

What's next for SLFYcare

We hope to improve our emotion recognition model by training it using patients with anxiety, depression and other mental health concerns and integrating more physical health data to better predict emotional distress. As more data gets accumulated, we hope to improve the accuracy of our machine learning models.

Share this project:

Updates