Fusion

Inspiration

This project was inspired after I lost my best friend to suicide. We realized only afterwards that he was suffering from a major form of depression that ultimately was too much for him to bear. But, he isn't alone: in fact, more than 50% of people who commit suicide suffer from some form of major depression (anxiety/mood disorders, schizophrenia, etc). The biggest problem, we've found, is that most people who suffer from these disorders have no clue that they're suffering from them because society hasn't really been in favour of us analyzing our own mood swings and emotions.

What it does

Fusion AI is your personal Mental Health AI assistant that helps people keep track of their psychological states over remote "sessions". Each session is divided into:

1. A video session where the user switches on their webcam which clearly shows their face, and carries on with their normal, daily tasks (such as reading an e-book) on the computer. In the background, Fusion AI tracks the user's facial emotion responses and generates an overall result of their emotions and mood swings.
2. An audio session where we derive further insights into the user's psychological state via their vocal sentiment. This helps Fusion to understand the user's emotional state deeper.

How we built it

This entire project is written in Python with:

1. TensorFlow powering the emotion recognition model
2. A Dash and Flask integration that powers the frontend, and backend respectively.
3. Firebase, where real-time emotion responses are stored
4. AssemblyAI, which powers the vocal portion of a Fusion AI session (including sentiment analysis)

Challenges we ran into

This was my first time working with databases, so there was a lot of back-and-forth between bug fixes and solutions. Additionally, training (and tweaking) the Tensorflow model took up a big chunk of my time which limited the number of features I could implement for this project.

Accomplishments that we're proud of

• Building a custom TensorFlow model from scratch using only the help of a published Paper for the model's architecture
• Creating an awareness platform that I hope will help others keep track of their emotions and mood swings

What's next for Fusion AI

I plan on working on a more robust approach to analyze someone's overall psychological state. My solution currently has a lot of sway and is easily affected by external factors that can cause results to skew in different directions. My plan is to address these issues and then publish this app to a wider market where I hope more people (students and working professionals alike) will benefit from this.

Built With

• ai
• css
• html
• javascript
• python
• tensorflow
• typescript