Halo - Virtual Companion

Inspiration

The idea behind Halo - Virtual Companion emerged from the increasing global issue of emotional well-being, especially in today's fast-paced world where loneliness, stress, and mental health challenges have become common. Inspired by recent advances in emotionally intelligent AI and technologies like Hume AI, we wanted to create a personal companion that could provide real-time emotional insights and support to users in their home environments. Halo aims to help users reflect on their emotions, providing meaningful insights and fostering a sense of connection by offering conversations tailored to emotional needs.

What it does

Halo is a virtual mental wellness companion that uses AI to detect a user's emotions through conversations. It tracks emotional history, provides personalized insights, and even recommends music based on the user’s mood. Halo can recognize when a user enters the room, initiate a conversation, and summarize daily emotional experiences using AI-powered tools. The app visualizes this data to help users track their emotional well-being over time and receive actionable insights on improving habits, aiming to offer a comforting presence when they need it the most.

How we built it

We built Halo using a combination of advanced technologies:

Hume AI: To detect and analyze emotions in real-time during conversations.
TensorFlow & COCO-SSD: For object detection and motion tracking, which recognizes the user when they enter the room.
Gemini API: To summarize conversations and emotional trends.
React: For building the user interface of the web app.
Spotify API: For generating personalized music recommendations based on the user's emotions.
Phoenix Tracing: For tracking how accurately the AI processes and summarizes the emotional data.

Challenges we ran into

One of the biggest challenges we faced was integrating various APIs and ensuring that the emotion detection was both accurate and seamless. Handling real-time emotional data in a way that felt natural while respecting privacy was a technical hurdle. Another challenge was optimizing the AI to not only detect motion but also provide insights from the captured conversations . Additionally, ensuring that the dashboard visualization was both insightful and user-friendly within a short timeframe was a major task.

Accomplishments that we're proud of

We are proud of how well Halo integrates multiple technologies to form a seamless experience. We successfully created a system where users can receive real-time emotional insights and reflect on their daily emotional trends. The music recommendation feature, which tailors playlists based on emotions, turned out to be an exciting feature that adds significant value. The sleek user interface and data visualizations for tracking emotional history were also highlights of the project. We are very proud of the final outcome and how many technologies we were able to integrate. It was all of our first time using computer vision in a project, and also our first exposure to some of the tools like Arize and Hume. Not to mention, half of our team are first-time hackers!

What we learned

Throughout the development of Halo, we deepened our understanding of emotion recognition technologies and how they can be applied in real-life scenarios. We learned how to effectively integrate complex APIs like Hume AI, Gemini, and Spotify, while navigating the challenges of real-time data processing and analysis. Collaboration within a tight timeframe helped us refine our project management, communication, and technical debugging skills.

Additionally, working as a team significantly improved our version control skills. We learned to efficiently handle merge conflicts, manage pull requests, and maintain clean, organized code through frequent reviews and continuous integration. These experiences taught us how to work more effectively in a collaborative development environment.

What's next for Halo - Virtual Companion

We are excited about the future of Halo and have several ideas to enhance its emotional intelligence capabilities. We plan to incorporate additional AI features such as mental health exercises and personalized advice based on emotional trends, while refining our emotion detection algorithms to improve accuracy. We envision expanding Halo to serve more user personas, such as seniors living independently, where mood tracking could be shared with family members for added support. Additionally, we plan to implement a Med Gemini-based AI assistant for deep conversations triggered by specific emotional cues, offering scientifically-backed mental health advice. As we continue developing Halo, we hope to scale it to support a larger user base, explore integrations with health apps and services, and enrich user experience with more AI-driven insights, making a meaningful impact on emotional well-being.