An unfortunate yet common reality for patients with memory loss (whether from Alzheimer's, dementia, amnesia, etc.) is the pain of forgetting the faces of the people in their lives—especially the ones that they love. Our goal was to use the power of AR and AI to help patients recognize faces, and to also track their memory over time. That's why we sought to augment their recognition with Recogvision.
What it does
Recogvision is an augmented reality view for patients with memory loss that helps them remember and keep track of the people they meet. When meeting someone for the first time, our smart voice assistant picks up on their conversation and uses natural language processing to infer the person's name. Their name, along with a photo of them from our camera are stored into our database. Later, if the patient sees this person again, they are given the option to display their name on the screen. Hopefully, the patient remembers the person's name. But if they forget, they can click a button to view the name. Recogvision also tracks the number of times a patient is pressing this button—or the number of times they forget a name. This statistic over time provides valuable insight into the progression of the patient's memory, and it can be reported to medical professionals as an expedited report.
How we built it
We built the frontend of our cross-platform application in Flutter. This app communicates with our Flask server hosted on Google Cloud, which stores and retrieves information from our Firebase database, as well as our very own person identification library we made in python.
Challenges we ran into
We ran into A LOT of issues while working with Flutter, especially related to cropping images from the video stream in real time. We also had some cacheing difficulties with Google Cloud that we managed to work out.
Accomplishments that we're proud of
We're really proud of our voice assistant and our person identification! These are no doubt the two most technically complex aspects of our project, and we are proud that we were able to use such high-level technology to create an intuitive, accessible, and functional solution for people who need it.
What we learned
We learned a lot in this project! Sohit and Anthony learned about video processing in Flutter, Sarthak learned about creating a computer vision library, and Dean learned about building backend servers and databases.
What's next for Recogvision
We want to take this idea further! We think this is a very effective way to help patients with memory loss, and we want to incorporate more features into the app such as showing the person's relation to you, information about them, or details about when you first met them.