Harmony AI

Inspiration:

The theme for UofTHacks X was about exploration and the escape from the pandemic, trying to innovate for our future. During the pandemic, many people were cut off from human interaction, therefore leading many to mental health issues. We wanted to solve this. Many don't have people that they are comfortable expressing their feelings, and we want to provide a 24/7 free, private, and helpful service to combat this.

What it does:

Harmony AI is a cutting-edge speech-to-text therapy bot that is designed to provide mental health support to individuals who may be struggling with the isolation and stress caused by the pandemic. The bot is trained to understand and respond to spoken language in real-time, allowing users to have a more natural and conversational experience with the AI.

The bot uses advanced machine learning algorithms to analyze the user's speech and respond with appropriate therapeutic prompts. These prompts are designed to help users express their emotions, understand their thoughts and feelings, and provide guidance on how to cope with stress and mental health issues.

How we built it:

Harmony was built using co:here's API which provided us with API keys to output trained data. We used Estuary's API keys to retrieve data from the user and send it to co:here where it would use the data to generate appropriate output. The Google Speech To Text API was used to catch speech and convert it to text for the data to be processed and outputted trained data from co:here in speech and text form. This strategy increases inclusivity amongst different group types such as those with hearing disabilities.

Challenges we ran into:

Caching and Gateway errors: One of the main challenges we faced during the development of Harmony AI was managing caching and gateway errors. These errors occurred when the system tried to retrieve or process data from the user, and it slowed down the performance of the system. We had to troubleshoot and find solutions to these errors, which required significant time and effort.

Integrating multiple APIs and technologies: Another challenge we faced was integrating different APIs and technologies to create a functional and user-friendly system. We had to make sure that all the different components of the system were working together seamlessly, which required a lot of testing and debugging.

Ensuring data privacy and security: As the system is storing sensitive user data, we faced challenges in ensuring that the data was being stored and processed securely. We had to implement different security measures to protect the user's data and comply with data privacy regulations.

Overall, these challenges required a lot of time, effort, and problem-solving skills to overcome, but they helped us to improve our technical skills and understanding of the different components of the system.

Accomplishments that we're proud of

Being able to speak to an AI and receive responses completely in speech: This is a significant accomplishment for our team as it demonstrates the functionality of our speech-to-text therapy bot. We were able to integrate different APIs and technologies to create a system that can understand and respond to spoken language in real-time. This feature allows users to have a more natural and conversational experience with the AI, making it more accessible and user-friendly.

Successfully implementing co:here and Estuary API's: We were able to use the API keys provided by co:here and Estuary to retrieve data from the user and generate appropriate output. This allowed us to create a system that can understand and respond to the user's emotions and provide therapeutic prompts.

Overall, we are proud of the functionalities and features that we have implemented in Harmony AI, which we believe will help to combat mental health issues caused by the pandemic.

What we learned

What is and how to use a speech-to-text API: We learned about the different types of speech-to-text APIs available and how to use them in our project. We learned about the different features of speech-to-text APIs, such as speech recognition, voice commands, and transcription of audio recordings. We also learned about the process of converting spoken language into written text and how to use the API to analyze audio data.

How to use co:here and Estuary API's: We learned about the different functionalities of co:here and Estuary API's and how to use them in our project. We learned about how to use the API keys and how to retrieve data from the user and process it to generate appropriate output.

How to use web socket connections for streaming audio: We learned about the process of streaming audio using web sockets and how to use it in our project. We learned about the different types of web socket connections and how to use them to send and receive data in real-time.

Overall, we learned about the technical aspects of building a speech-to-text therapy bot and how to integrate different APIs and technologies to create a functional and user-friendly product.

What's next for Harmony AI:

Our vision for Harmony AI is to enhance its capabilities and make it a more comprehensive and inclusive tool for mental health support. To achieve this, we aim to implement a variety of voices and support for multiple languages to cater to a wider audience. Additionally, we plan to integrate machine learning algorithms to improve Harmony AI's understanding of the user's emotions and provide more personalized therapy sessions. Building a system that can store data and pick up from the last time the user left it is something we tend to implement so that the user doesn't have to start their session from the beggining. Furthermore, we will strive to expand Harmony AI's coverage of mental health topics to ensure that it can effectively address a diverse range of issues.