MemoryMate

“Compassionate care for when every memory matters.”

Comment

Inspiration

Nearly everyone at some point in their lives has met someone suffering from dementia. Two out of three of our teammates have lived with relatives suffering from the disease and have seen first-hand the deteriorating quality of life of patients and demanding needs of caregivers. Our team was moved to create a solution, MemoryMate, which increased quality of life for patients with dignity and compassion while democratizing access to effective home healthcare in the later stages of life.

What it does

MemoryMate interprets conversations between patients and caregivers in real time to extract key points like names, dates, and people. The device employs a unique, multimodal model to identify the state of the patient and whether intervention from a caregiver is required, intervening itself in most situations by providing a compassionate and caring response. In addition to supporting the needs of the patient, MemoryMate offers real-time summaries to care providers and the ability to schedule reminders to patients, decentralizing healthcare and allowing caregivers to make more informed decisions regarding patient outcomes.

How we built it

In order to achieve real-time audio processing, MemoryMate uses a combination of Google Cloud applications and OpenAI’s Whisper to parse speech-to-text and filter out important information. In order to identify the current needs of the patient, we employ a novel combination of emotion detection through Hugging Face’s BERT as well as phrase detection identified by caregivers. When intervention is deemed necessary, MemoryMate utilizes a hand-trained Gemini large language model built on years of conversation data between distressed patients and consoling therapists to craft consoling and compassionate responses as well as powering summary generation.

Challenges we ran into

Our biggest challenge when building MemoryMate was finding a UI which reflected the uniquely different needs of patients and caregivers while also allowing for the communication of large quantities of complex information. In the end we settled on a platform called Flet as we felt it best supported the project’s goals in the long term and provided the most cohesive and comprehensive interface for the project.

Accomplishments that we're proud of

While we’re certainly proud of the technical obstacle we had to overcome to see our project through, we are most proud of our project’s potential to be a driving force for good across the world. By providing compassionate and dignifying dementia healthcare in a flexible form factor designed with caregivers in mind, MemoryMate is excited about the future of democratizing access to healthcare.

What we learned

As a team, we learned much in nearly every facet of this project. Not only were we able to refine our ability to design a effective input system, but also gained experience tying that input together with a complex, generative AI-based processing system and uniquely-suited user interface.

What's next for MemoryMate

MemoryMate would like to design dedicated hardware for our project similar to a home assistant with emphasis on dementia home healthcare. Additionally, we plan to redesign our software stack to be split into a handful of applications which better reflect the unique needs of different users.

Built With

Share this project:

Updates