Memory Companion

Memory Companion
Automated call when Fall Detection
Alzheimer progression analysis
Object Tracking
Github codebase for the product

Inspiration

Driven by the heartfelt concern for the well-being of the over 50 million individuals worldwide affected by Alzheimer's and our deep empathy for the elderly members of our own families who require constant support, we embarked on a mission to develop a revolutionary system using technology. This system serves as a guiding companion for patients and elderly individuals, assisting them in locating misplaced items and offering essential emergency aid and aiding therapeutics and doctors by providing the track of a person's digital biomarkers.

What it does

Our solution provides a comprehensive care system designed to support and enhance the lives of elderly people and individuals affected by Alzheimer's, providing peace of mind for families and empowering patients to live with dignity and independence. We accomplish this with three different components:

Memory Companion The Memory Companion enhances patients' memory by assisting them in locating misplaced objects they may have forgotten. By maintaining a real-time view of the room and detecting object motion, the model utilizes stored memories to provide users with valuable assistance when they inquire via ChatGPT, helping them find the items they are looking for.

Automated call when Fall Detection Patients and elderly people often feel dizzy due to excessive medication and declining health and hence are more prone to facing accidental falls than normal. Our model detects falls in real-time and immediately alerts emergency contacts through phone call, (using Twilio), ensuring prompt treatment for patients and elderly individuals prone to accidental falls.

Alzheimer progression analysis Using the capabilities of the Hume API and leveraging the analytical behavior of ChatGPT, we provide a tool that assists doctors and family members in monitoring patients' behavior and tracking their emotions during interactions with the system (with user consent). Recent research has revealed that brain changes associated with Alzheimer's can begin up to 20 years before symptoms manifest.

How we built it

Our solution is rests upon three different pillars:

Large language model + Voice Integration The patient can interact with the AI bot using voice (whisper API) and these interactions are fed into LLM(GPT-4) which then takes in contextual information through the camera (in the room), understands the scene and provides answer to the query (location of certain object). The iot then provides a vocal answer to the query.

Indoor Object Tracking Keeps tracking object while the object is in the room and sends outputs to gpt-4 which then takes input through voice engine, derives the answers and provides answers back to voice engine to reply back.

Data Analytics and Emotion detection By utilizing data from various sources and Hume API, we construct a digital history of potential patients, enabling early detection of the disease. The patient's search trends can help doctors understand the progression of the disease. Additionally, we develop an interactive tensorboard API that captures real-time feed, serving as a valuable tool for doctors and family members to gain granular insights into the patients' lives.

Challenges we ran into

To accomplish our goal, we faced numerous technical and practical challenges. But with the use of sponsor APIs, we aced our way!!

Elderly and patients need a smart, personalized system to answer their queries in an easy-to-use manner. With the use of OpenAI’s whisper API and GPT4, we were able to accomplish this. Furthermore, we wanted to track emotions and behavior of users to provide insights. With the use of Hume API, we were able to track this and provide deeper insights to doctors.

Accomplishments that we're proud of

We are proud that we built a more comprehensive and impactful product than what we had in mind when we entered that door of Martin Luther King Building. We believe our biggest achievement is that we made this product with capabilities of making into the market, creating impacts in the healthcare sector as well as improving and helping elderly people with Alzheimer.

With the memory companion, we can rest assured that there is someone to assist our elderly and patients. Apart from assisting them in finding their things, we can also rest assured if they fall or need emergency assistance, the product will perform fall detection and automatically call the family. Furthermore, we are happy that the solution provides insights that would even help doctors in providing better care.

What we learned

During our interactions with fellow hackers, and sponsors, we learnt about new technologies, APIs and capabilities of GPT4, LLMs and products like hume, mindsdb etc, to make our life better. Furthermore, In the Meetup village, we were introduced to Roger. We learnt how Berkeley Sky fund is so much invested in making patient lives better. We gained deep insights from them on how we can scale our product and deliver it to the market.

What's next for Memory Companion

Technical enhancements: Currently, we offer a proof of concept (POC) for our product utilizing existing AI tools and the sponsor's API. Moving forward, we aim to fine-tune the models specifically for the indoor settings to optimize performance for our targeted use case.

Market research analysis: After conducting our preliminary research, we have observed the effectiveness of the proposed and implemented solutions for patients with Alzheimer's and elderly individuals. Looking ahead, our focus is to identify potential app users and establish collaborations with doctors to validate the utilization of our digital history in their diagnostic processes.

Scaling: With the feedback from the judges and audience, we would like to build an app for this which scales to millions of users.