Happiness-Seeker chatbot

A human-bot connection to lift your mood

Comment

Inspiration

In 2020 due to the coronavirus pandemic, our lives have completely been changed. Since I was finishing my graduation from Arizona State University, I was living in the US away from my family who couldn't travel to the US for my graduation. It's very difficult to express emotions to people who might also be feeling sad because of the same situation. Also, due to the pandemic, the job market has become static and I've been without a job for months now while despairing over my visa situation. During this time, the best thing to do is to keep oneself busy and that is what I've been trying to do in recent days. However, it's not easy to come up with new activities while being in a state of despair and anxiety. It then occurred to me that having an app that can suggest actions or activities that I can do based on what I am feeling would be extremely useful, particularly for people who are in a similar situation as myself. Almost all recommendation systems are based on previous likes and consumption, however, getting suggestions based on your current emotion has a sense of intimacy which is not available in existing systems. Then I found about this hackathon and managed to make my friend from India convinced about the idea and thus, began the journey for the happiness-seeker bot

What it does

Happiness-seeker is a chatbot that uses the powerful Natural Language Processing power of Wit.ai to understand the emotion of the user from their responses and then suggests useful actions to either lift the user's mood when it's down or suggests ways to enjoy or celebrate their achievements. It can suggest nearby restaurants, movies, music, vacation plans, or nearby activities based on the mood. It has more personal actions suggestions like calling your partner, speak to your family, speak to your friends. It can also detect medical emergencies and suggest nearby hospitals or emergency rooms. The moods are detected using a custom build entity - get_emotions (wit.ai) and are validated further using built-in trait wit$sentiment. The emotion can range from extremely positive, positive, neutral, negative, and extremely negative. Each range of emotions have their own action items and the top 5 five choices for the particular emotion is chosen and displayed in the carousel of the generic message template. For the purpose of the hackathon, the choice of actions is limited and the attached links direct to their respective Facebook with the exception of music, where it opens the Spotify app. The user can then decide which particular action to choose to continue their online activity. The user can also re-speak their emotions to get better choices. Currently, the app supports both voice-based and text-based interactions with the messenger platform.

How I built it

The app was built leveraging the messenger platform and the webhook capabilities of a Facebook app. We created a custom made a Facebook page that served as the UI for the chatbot. We connected it to a Facebook app using my developer's account. Then we added the messenger and webhook capabilities to the app. We built the app using python and flask. We added the necessary libraries, dependencies required to handle the voice interactions. The app doesn't connect to any database and hence doesn't store any user's information. It only keeps the voice attachment temporarily for the wit.ai's speech API to process the information. We trained the wit.ai app by first collecting the common utterances for a wide range of emotions from positive to negative. We spent time manually annotating the entities and traits. After the training was completed, we used the Wit.ai speech and message APIs to communicate with the app. The interactive nature of the app was maintained by the pymessenger bot which took care of structuring the payload for request and response. As the app was developed the utterances encountered by the app increased and they were retrained and validated.

Challenges I ran into

Almost all the things that we encountered to build this app was completely new to us. We started learning from scratch about various kinds of web and mobile app development. Ultimately, based on the time frame, this was the most pragmatic approach that we found for the purpose of the hackathon. The biggest challenge that I faced is to understand the process of handling audio files within the system. Android records the audio as mp4, which was not recognized by the wit.ai speech API. We then converted the .mp4 file to .wav file which ultimately worked. Another issue was with making our app public. It requires a privacy policy URL which we didn't have much idea about. However, almost all the challenges were an opportunity to learn more about the functionalities and features of wit.ai.

Accomplishments that I'm proud of

The biggest accomplishment was to integrate the voice attachments with the text-based system. We didn't have much idea about handling audio recording and hence it was quite a challenge. However, the greatest reward was when the app accurately recognized our utterances and gave the correct emotions and results.

What I learned

There are plenty of things that make this hackathon a huge learning experience. Firstly, we delved into a completely unknown territory of app development. We learned a lot about handling API requests, responses using python and flask, connecting webhooks, access apps through webpages, etc. Secondly, we got exposed to the wit.ai platform and all the fantastic functionalities it offers. We also learned about full-stack development and what it takes to upgrade a feature or add functionalities to existing systems. Lastly, we learned a lot about handling different kinds of files in python.

What's next for Happiness-Seeker chatbot

The future plan for the chatbot is to add more functionalities. Currently, there is a limited choice of actions for particular emotions, the action pool can be further extended. The current version has limited voice interactions, in the future callback functionalities can be added for the button template such that it can be interacted by voice commands. A better way of choosing the actions can be done by storing the user's preference when they encounter a particular emotion. It can also be extended to detect the emergency situation and can provide help to the user by contacting first-responders and emergency contacts.

Share this project:

Updates