Remembrall

API
Github
Location
Patient Data

Inspiration

It’s a challenge becoming a “parent to your parent”. We know, one of our teammates has been there. A member of our team has personally experienced the impact of dementia within their family, with both their paternal grandmother and maternal grandfather affected by the condition. Tragically, his grandmother, during an episode of disorientation, lost her way to their family home and found herself at a bus station, where she was robbed of her gold bangles. Upon our research we found that African Americans have the highest risk of dementia— 38 percent according to a study over a 25-year period beginning at age 65. As Engineers striving towards using our skills for the Social Good, we aim to reduce the pain of Dementia on both the caregivers and the people diagnosed trying to make their lives a little bit easier.

What it does

Remembrall is a conceptual design for a wearable device tailored to address the specific requirements of individuals with dementia and their caregivers.

Facial Recognition: This feature allows individuals with dementia to identify familiar faces through facial recognition technology, storing essential information such as names, relationships, and the last encounter in a database. Caregivers initiate the setup by inputting basic information about the patient and teaching the device to recognize key family members. The built-in camera aids in recognizing and recalling faces, offering users the choice to add new acquaintances to their database.

Live Tracking and SOS: Besides facial recognition, the device includes location tracking accessible via an administrative dashboard. This functionality enables caregivers to monitor their loved one's whereabouts, with options to directly contact emergency services or initiate a call if they sense danger. Additionally, the device features an SOS button for individuals to press in distressing situations.

Thought Logs: The wearable device captures audio snippets at regular intervals, which are then transcribed, summarized, and stored as logs. This allows users to revisit their thoughts, tasks, and activities, aiding memory recall.

Task Checklist: Caregivers can send a daily checklist of tasks to the wearer's interface, helping individuals remember and complete daily activities.

Community Chat: The platform provides a secure environment for both caregivers and those with dementia to share experiences and foster a supportive community.

Through these features, Remembrall aims to enhance the lives of those affected by dementia, offering them a sense of independence and support.

How we built it

A graphical user interface (GUI) has been developed in Python to simulate the functionality of an advanced wearable computer, akin to the Apple Vision Pro, showcasing a range of features designed with sensitivity to the needs of individuals experiencing dementia. The team utilized the OpenCV library for the facial recognition feature, where the model was trained by capturing 200 images in a span of 30 seconds and subsequently labeling the identified faces. To support location tracking, a Python-based location feature was integrated, enabling the calculation of the distance between the user's home address and their current location. For capturing Thought Logs, a Python library was employed to record audio at intervals of 30 seconds, with these recordings being converted from speech to text. The GPT-4 API was then used to summarize the text, extracting the core context of each sentence. This thoughtful approach aims to assist individuals experiencing dementia in keeping track of their thoughts, tasks, and priorities, thereby supporting their independence and quality of life.

Challenges we ran into

One of the most significant hurdles encountered was the adaptation of the solution for use with advanced smart glasses, such as Google Glasses and Apple Vision Pro. Given the high cost of these devices, the team opted to construct the software aspect of the solution within an emulation GUI. Additionally, integrating the various modules presented challenges. These modules had been developed independently, leading to some difficulties during the integration process.

Accomplishments that we're proud of

The team takes great pride in the multitude of features that were managed to develop within a limited timeframe. We are equally proud of our ability to create a fitting adaptation of the solution for the GUI, serving as an effective representation even without access to the actual glasses. Moreover, the successful integration of all the independently developed modules is another accomplishment that brings the team immense satisfaction.

What we learned

During this hackathon, we embarked on a journey of learning and discovery. We mastered the art of text summarization by leveraging Generative AI technology, and diving deep into its capabilities. We delved into the principles of User Design, Design Thinking, and User Experience, enhancing our approach to crafting solutions. Furthermore, we sharpened our skills in modular programming, learning to integrate a variety of modules seamlessly to forge an end-to-end solution. This experience enriched us, broadening our knowledge and skills in ways we hadn't imagined.

What's next for Remembrall

The most significant advancement will be bringing the solution closer to reality by developing it on Smart Glasses or their SDKs, as soon as they become available to us. Another step with Remembrall involves creating a fully functional community section.