VibeCheck

Splash screen
Text analysis splash
Text analysis result
Image analysis splash
Image analysis webcam screenshot
Image analysis bar graph

Inspiration

Through quarantine, we have noticed that communication has become super tough and it has become very easy to fall into the habit of assuming other people's thoughts and overthinking the intent behind messages. We agreed that it must have been even harder for people who already struggle with reading emotions in person (such as people with ASD) and thought we'd try to find some way to help.

What it does

Our project conducts sentiment analysis using Google Cloud's Natural Language API. It then displays the analysis through a simple interface.

How we built it

We built the front-end application using React and Node.js. The front-end libraries used are: Chart.js, React-webcam, and MUI-core.

The back-end is a Flask python server that access GCP's NLP API. We also used Keras for Tensorflow to train a popular neural network for image processing to classify facial expressions (VGG16). We trained it on the FER2013 dataset from Kaggle. The backend uses OpenCV to process and extract faces to feed to the neural network.

Challenges we ran into

We're normally used to sending text in request bodies, so we had to learn about communicating visual data through base64 encoding. This also gave us the problem of decoding base64, which is done in the back-end server.

Accomplishments that we're proud of

We are quite elated that we were able to create this application on time. We aren't used to using base64 or GCP, so we are glad to have completed this project.

What we learned

We learned about the importance of streamlining our taskflows so that we have the parts necessary for everyone to proceed with their work. Since this was our first online hackathon, we used a Trello to organize our tasks so everyone could see at any time.

What's next for VibeCheck

Given more time, we would have implemented an audio sentiment analysis front-end. We attempted this but simply did not have enough time. We would also like to extend this to a communication platform such as video chat or messaging since this would be much more beneficial to our target audience's mental health.