Growing up, we heard about our friends' grandparents suffering from Alzheimer's and forgetting about their loved ones – until we also had to witness our grandparents going through such an ordeal.

This application is dedicated to those who are suffering from memory loss, and for everyone around them who cares.


This HoloLens application creates visual cues and allows users to interact with the application to remember daily routines in the lives and their loved ones. Whenever the application asks the user about what they're doing, the user can update the application about what they're doing, BUT if the user doesn't remember what he/she is about to do and says that out loud, the application scans the environment for clues and tells the user what they were thinking about doing, and if they've done it recently.

The application also has a holographic dashboard that users can see when they look up! It features picture frames of the user's loved ones, along with their names and their relationship with the user. This is a constant reminder about the important people in the user's life who cares about him/her so much.

There's also an alarm system that prevents users from wandering too far off. Whenever the user crosses certain boundaries created by markers on the spatial map, an alarm will trigger on the HoloLens, telling the user to come back!

We're also using Twilio to keep a log of what the user is doing! Every time the user takes some action, like taking medication, the application will send a text to the user's phone. If the user wanders off and crosses the alarm, a text is sent to a friend or a family, warning them about the user wandering off.

How we built it

We used Unity to create the AR experience and deployed it onto the HoloLens. We trained our machine learning model on Microsoft Custom Visions in Cognitive Services to recognize key objects in our daily routine (pill bottle and bodywash). We used a Microsoft Bot and Twilio to text the user events that they've done .

Challenges we ran into

We also tried using Twilio to send a list of events that the users have done (such as taking medication and the time stamp) and give family members a heads up if the user wanders off, crossing the "dangerous" boundary. Twilio was successful in sending a SMS message on Unity locally, but failed sending messages when we deployed it onto the HoloLens. Our solution was to use a Microsoft Bot to receive information from the HoloLens and calls the Twilio API for us, essentially functioning as our middleware for this functionality. And also, two members out of three didn't have much experience with Unity and the HoloLens.

What's next?

We're going to add more routines that our dataset is able to recognize, and more visual cues for users to remember about their surroundings. Instead of having a line of pictures in the air, it would be great to create a flip book that instantiates on command, and the user can flip through it with gestures.

Built With

+ 2 more
Share this project: