Elevator Pitch: Recall helps dementia patients identify faces, recall past conversations, and practice active recall through memory games, restoring confidence and dignity, one interaction at a time.
Project Story:
Inspiration
More than 50 million people around the world are living with dementia, and the earliest stage offers the greatest opportunity to make a difference.
One team member has personally experienced dementia in her family. We’ve witnessed the heartbreak of loved ones struggling to recognize the people who matter most to them. That experience inspired us to build an app designed to support individuals in the early stages of dementia—helping them preserve memories and stay connected to what matters most for longer.
What it does
Our app is designed to help individuals with dementia recognize and recall important people in their lives. Using a camera-based recognition feature, the app can identify familiar faces and provide contextual reminders about each person, including relationship to the patient and past conversations.
Our app is meant to be a mobile prototype, allowing users to identify people in real time, but in the future we would like to expand to Augmented Reality glasses as it provides a more seamless experience for the patient. We were limited to mobile due to hardware constraints. When a user points the camera at someone they know, the app recognizes the individual and their relationship, and what they last discussed together.
The app also includes a recording feature that allows users to save conversations for later review. These interactions are stored and used to generate personalized quizzes that help reinforce memory and strengthen recall of important relationships and life details.
In addition, we provide a caregiver dashboard where caregivers can add, update, and manage important people in the user’s life, ensuring that the system remains accurate and up to date.
Overall, our goal is to create a meaningful and supportive tool that helps dementia patients stay connected to the people and memories that matter most.
How we Built It
The backend runs locally on a Mac using Flask for the API server and InsightFace (ArcFace model) for fast, accurate face embeddings. Face matching uses cosine similarity to identify enrolled people. On the iOS frontend, we built a SwiftUI app with real-time camera integration for face recognition and live audio recording. The app uses on-device speech recognition (iOS Speech framework) to transcribe conversations without sending audio to external servers. For conversation summarization, we integrated Ollama running Mistral 7B locally, allowing users to get AI-generated summaries of conversations entirely on their own hardware. This approach ensures sensitive personal data never leaves the user's network while maintaining fast, reliable performance across all features.
Challenges we ran into
Apple doesn't provide a built-in face recognition API beyond basic face detection, so we initially explored CoreML models for on-device processing. However, we did not have success finding them. Rather than compromise on accuracy or privacy, we pivoted to hosting InsightFace (ArcFace) on a Flask backend server running locally. This hybrid approach gave us effectiveness on the backend while keeping the iPhone app lightweight.
Accomplishments that we're proud of
- We protected data privacy by running everything locally.
- We worked around having to pay for cloud services by running everything locally and worked around having to pay API services by using open source LLM models.
- Hand-drawn background for UI
What we learned
Learned how to create a mobile app using Swift and XCode. Learned how to integrate LLMs and facial recognition models. Learned how to troubleshoot errors in Xcode and how to run our program on mobile devices. Learned how to connect Flask backend to swift frontend locally
What's next for Recall
Next, we plan to integrate the app with smart glasses so users can receive real-time, hands-free information about the people they see. This will allow them to instantly and easily identify their loved ones and recall their most recent conversations without needing to use a mobile device.
Log in or sign up for Devpost to join the conversation.