COVID-19 is one of the most traumatic events in recent memory. These kinds of life-changing experiences often have a lasting effect on our mental health. With most of us in lockdown and separated from friends and family, many of us don’t have our support networks to rely on. Too often, our mental health can start deteriorating without us knowing. The biggest barrier for many people is simply reaching out. Even though that sounds easy, it can be difficult overcoming natural and social barriers in place.

We wanted a service that would take the initiative and ask you how you're doing. We wanted a way to offer encouragement and advice, and also a way to track how you’ve been feeling. We decided that Voice Assistants would be a great way to offer this - every day when you get home, Alexa asks you how you've been. More people than ever have Amazon Echos or Google Homes in their household, with a 65% increase over the past few months. While they can’t ever be a perfect replacement for friends and family, we can leverage voice assistants to check in us on a daily basis.

What it does

ketchup is a custom-built Alexa skill that asks the user how they’re feeling everyday. You can either trigger it by saying "Alexa, let's catch up", or by creating a routing to be asked everyday at a certain time. It uses sentiment and emotion analysis to determine their mood, and stores that information in a CockroachDB database. Based on the user’s current mood and how they’ve been feeling for the last two weeks (along with slope, r2 calculated by a linear regression model), ketchup will offer custom responses and suggest resources to aid the user. In the case of self-harm or suicide risks, ketchup will suggest the user reach out to the National Suicide Helpline.

How we built it

We built this in two general teams - front-end (Alexa), and the back-end. On the front-end, we have ketchup as an Alexa skill, built with Node.JS on a lambda function.

The backend is built with flask, and is hosted on Google-Cloud in an Ubuntu 18.04 instance. We use Google-Cloud-Language API along with DeepEffects emotion analysis API to retrieve sentiment/emotion information, and we store all this information in CockroachDB with a SQLalchemy wrapper.

Challenges we ran into

Debugging Alexa is a nightmare. There aren't too many helpful error messages, and continually listening to voice responses can be frustrating. It also wasn't a technology that many of us were familiar with. We also decided to use some new technologies like CockroachDB, and making sure that all the contingencies interacted with each well was a big challenge. Another challenge was that we wanted to be sure that our responses were helpful and would not cause harm. That meant that we had to spend a lot of time thinking about them, not just to create a natural flow of dialogue with Alexa, but also to provide them with meaningful responses based on their current and previous moods.

Accomplishments we’re proud of

We are proud of our ability to overcome working virtually to come together to brainstorm, ideate, and develop an idea to create a working prototype. It wasn't always easy, but we worked well together and finished a working project.

What we learned

From this project we learned how to establish a server with flask and how to work with CockroachDB. We learned that it is important to have the interaction between the user and Alexa feel natural as this is imperative for the user to feel comfortable sharing their feelings with Alexa and have ketchup perform as desired.

What’s next

For future iterations we plan on adding a feature that allows for sentiment or emotion analysis of the user's voice, including factors such as tone, inflection, and sarcasm, to get a more accurate picture of the user’s current mood. Additionally, we would like to improve Alexa’s current responses to a user’s mood by talking to therapists and psychologists and consulting online resources. ketchup was built with the Alexa Developer Console and Google Cloud Console. We also utilized Google Cloud’s Cloud Natural Language API and DeepAffects’ Text Emotion Recognition API.

Built With

Share this project: