CompanionAI

Example conversation with an elderly person
Example conversation with images

Inspiration

The inspiration behind this project stems from personal experiences with our elderly grandparents, who often struggled with loneliness, noticeable cognitive decline, and memory issues. We wanted to create something that could provide both emotional support and practical assistance to those facing similar challenges. The idea of an emotional AI companion felt especially meaningful because it could help bridge the emotional and social gaps that many elderly people face, offering comfort and engagement in a personalised manner.

What it does

This project leverages Mistral AI models to create an interactive emotional AI companion, aimed at supporting the elderly and those with memory challenges. The AI Companion uses the Pixtral-12B model to process chat-based queries and image-based descriptions. Additionally, we have integrated a Telegram Bot for easy and personalized interactions.

Features:

Text-Based Chat: Interact with the AI using natural language. The bot is designed to provide empathetic and context-aware responses to support users.
Image Analysis: Get AI-powered descriptions of images. The bot can analyze photos, such as old family pictures, to provide descriptive and empathetic insight.
Memory Integration: The bot maintains an ongoing conversation history as well as over multiple sessions, allowing for more personalised interactions.
Telegram Bot: The CompanionAI is also available as a Telegram bot for easy access and direct conversations on any device. The bot can handle both text and images seamlessly.
Dynamic Name & Background Setting: Users can dynamically set their name and provide additional background information to make responses more personalized.
Voice Interaction: Voice-to-text functionality is available for text-based interactions through the terminal, providing enhanced accessibility.

How we built it

The project leverages Mistral AI models, specifically the Pixtral-12B model, to create an interactive, emotional AI companion. We aimed to provide support for the elderly and those with memory challenges through a combination of text, voice, and image analysis.

Pixtral-12B Model: This model processes chat-based queries, providing empathetic and context-aware responses.
Prompt engineering: We created a prompt that nudges an LLM to provide more empathetic responses than it would otherwise.
Image Analysis: The AI provides descriptions of images, such as old family photos, which can trigger meaningful memories for elderly users.
Telegram Bot Integration: The AI Mind Companion is accessible as a Telegram bot, allowing users to interact with the AI easily on their preferred devices. This bot handles both text and images seamlessly.
Voice Interaction: For added accessibility, we incorporated voice-to-text and text-to-voice functionality, allowing users to speak their questions and receive responses as well. This was done using open-sourced speech-to-text and text-to-speech libraries.
Dynamic Personalisation: Users can provide their name and background information, which the bot uses to personalise responses, making each interaction feel more relevant and considerate. Furthermore, all the conversational history is stored and then used as a We used RAG for this.
Empathetic Dialogues Dataset: Additionally, we fine-tuned Mistral Large using this dataset to improve the AI's ability to engage empathetically (API fine-tuning for Pixtral-12B is not yet available).

Challenges we ran into

Building an AI companion that can effectively support the elderly came with its unique set of challenges:

User Accessibility: Balancing technological capabilities with user accessibility, especially for elderly individuals who may not be comfortable with technology, required thoughtful design choices.
Model Fine-Tuning: Fine-tuning the Pixtral-12B model was limited by the current availability of API-level fine-tuning options. We had to adjust our approach to achieve the desired conversational quality without dedicated fine-tuning.
Integration Across Modalities: Combining text, image, and voice interactions seamlessly within a single bot interface (Telegram) took significant time to ensure a smooth user experience without overwhelming the user.
Empathy in AI: Training a model to understand and respond to emotional cues was complex. We used the Empathetic Dialogues dataset to improve this aspect, but replicating genuine empathy remains a challenge.

Accomplishments that we're proud of

Empathetic Interaction: We were able to create an AI that could hold empathetic, emotionally supportive conversations, which is crucial for elderly users dealing with loneliness.
Seamless Integration: Integrating the AI into Telegram allowed us to provide easy access for our users on any device, which helps ensure that the tool is useful and accessible.
Image Analysis: Adding the ability for the AI to analyze and describe images, especially old family pictures, is something we believe adds a lot of value to the interactions and makes it more engaging for users.

What we learned

Throughout the development process, we gained a deeper understanding of:

Empathetic AI Design: Creating an AI that can recognize and appropriately respond to human emotions required thoughtful selection of datasets and training approaches.
User Accessibility: Developing features like voice-to-text taught us the importance of accessibility, especially for elderly users who may find traditional typing difficult.
Cross-Platform Usability: Integrating the AI companion into Telegram was a rewarding challenge that provided insights into the complexities of developing a multi-modal AI interface that feels natural and accessible.

What's next for CompanionAI

Enhanced Emotion Detection: We plan to improve the model's ability to detect and respond to emotions more accurately, making interactions even more empathetic.
Expanded Language Support: Adding support for multiple languages to make the CompanionAI accessible to non-English-speaking elderly populations.
Improved Memory Functionality: Currently, the AI maintains a conversation history, but we aim to develop a long-term memory that can help the AI remember user preferences over multiple sessions.