CogniCare

The CogniCare landing page.
A patient recording their POV with a camera.
3 generative AI-powered, cognitively stimulating exercises to assist memory retention.
(1/2) Situational questions for the patient to explain in their own words.
(2/2) Accessing the exact memory associated with the question.
Face-matching mental exercise. Generated with facial recognition data from your memories that day.
Name-matching mental exercise to build on the face-matching one.
Our open memory explorer, to semantically search through your memories with natural language.

Inspiration

Alzheimer's disease, a form of dementia, is a growing concern, with millions of individuals and families affected worldwide. The journey with Alzheimer's is challenging, often marked by a decline in memory and cognitive abilities. However, with the advent of digital technologies, there's a beacon of hope. We introduce 'CogniCare,' an app designed to aid those living with Alzheimer's in recalling and cherishing their life's moments.

What it does

Our app records daily interactions through a discreet camera, helping patients recall faces, names, and details related to the interactions. Users are prompted with images of people they met by selecting a date, encouraging them to remember names and interactions. This process aids in reinforcing memory, maintaining social connections, and reducing the stress and confusion often associated with memory loss in Alzheimer’s patients.

How we built it

We built CogniCare using a variety of AI and soft-dev tools. First, we use OpenCV to downsize the recorded videos and extract identified features. Through the video features, we recognize key frames using an oriented fast and rotated method. Next, through GPT-4's image embeddings, we embed the keyframes from our memories. We performed face recognition on the video features to identify the people contacted during your day. Using audio transcriptions, we use GPT-4 to combine key frame embeddings and the transcript for a holistic overview of the user's memories. Using these embeddings, we generate memory-stimulating questions and their corresponding correct answers. This allows us to gauge how much of their day the user remembers for memory diagnosis versus time.

The frontend was built with Next.js, styled with TailwindCSS and the Shadcn UI library. We created complex, well thought-out layouts to effectively reach the goals of our project, integrating interactions, video components, and much more.

Challenges we ran into

Size constraint of the videos as well as cleaning the audio for explicit speech-to-text conversion.
POV videos make it difficult for the face detection and recognition model to perform analysis.
Chunkwise separation of video based on the dissimilarity between frames using ORB features descriptors model.
- Creating visual interactions and integrating video components on the webpage using NextJS framework.

Accomplishments that we're proud of

To revolutionize patient recollection power, integrate a daily performance scoring system that challenges and tracks progress, combined with OpenAI's GPT AI technology for personalized, adaptive cognitive exercises and interactive experiences. This approach not only encourages consistent improvement but also ensures a highly engaging and tailored user experience.

What we learned

Participating in the 24-hour hackathon to develop CogniCare was an incredibly personal and enriching experience for our team. Throughout this intense and focused period, we learned so much more than just technical skills. The process taught us about the resilience and creativity required when time is limited and the stakes are high. We also gained a deeper understanding and empathy for individuals with Alzheimer's, which fueled our passion and dedication to the project. As we worked through the night, troubleshooting, brainstorming, and refining our ideas, we were constantly reminded of the real-world impact our work could have.