Hearing aids are extremely common. Glasses are extremely common. Even wheelchairs and other mobility devices are incredibly common. However, there is currently no accessible technology to assist those with memory loss or other neurodegenerative symptoms. Inspired by our understanding of the difficulties of Alzheimer's and dementia for patients and their loved ones, we decided to create a solution based in augmented reality.

What it does

Our technology takes in a video stream and provides two key benefits. First is machine learning powered identification of people and common items in the frame, which solves some of the most pressing emotional issues for those with memory loss. Second is auto journaling key daily events into a user interface for periodic review. This solution is end-to-end - it directly digitizes the two biggest methods to improve quality of life for patients - instant annotation and reminiscence therapy. Currently, both of these methods are performed without technology, as live-in caregivers physically annotate a household while patients are recommended to take periodic journals.

At an impact level, our solution extends beautifully. With digitized journaling and annotation, doctors can see exactly what the patient is experiencing. With tedious tasks accounted for, caregivers and patients can focus on emotional wellbeing. In the same vein, memory loss symptoms will take a lesser emotional toll on patients and their loved ones, especially as patients regain confidence that is key to their mental health. Finally, we see our tool as a great equalizer - while caregivers in some areas cost up to $75,000 a year, even an advanced AR headset can be had for a few thousand dollars.

How we built it

We have some truly beautiful code integration happening on three fronts. First, the product website is complete with original, custom artwork complemented by a pleasing interface. Second, the backend is powered by a robust machine learning model that we trained ourselves. Finally, integration across the platform is cleanly implemented through Flask and Google Cloud.

Ethical Considerations

The first comments we had when generating this idea surrounded the ethical implications. One of the biggest reasons for the failure of Google Glass was the ethical question of having a wearable camera on the person at all times. For this project, we pinpointed three critical ethical concerns with remembAR, and proceeded to address them in robust ways:

1) Holographic projections may confuse elderly citizens who are new to the technology. For example, a task imitation of making a cup of tea may confuse someone with Alzheimer's or dementia and cause an accident like a hot water spill. To counter this, we designed our project to be minimally invasive while retaining helpful suggestion value. This ensures that we are unobtrusive with our help.

2) We note that Alzheimer's and dementia, along with other diseases, are linked to poverty and ethnic minority populations, which in turn are generally linked with slower technology adoption. This is potentially ethically dangerous, given that AR is still a very new technology. However, we believe our technology actually bridges a socioeconomic gap. This is because firstly, the adoption curve for a relatively passive technology is shallower. These headsets are actually fairly intuitive, and we believe adoption will be similar to that of a touch screen. Secondly, this technology would fulfill many of the same tasks a live-in caregiver might, and is cheaper by a large factor. In some markets, caregivers demand up to $75,000.

3) The largest concern is certainly privacy, as is the case for any technology that requires a constantly active camera. Currently, our tool only identifies people that the model is already trained on. Future iterations could involve obtaining consent to be stored in the program's database, much like Google Photos obtains permissions to run face recognition algorithms on photos we store with them. In this way, family and friends would be most likely to oblige permission, and privacy of others would not be impacted as greatly.

Of course, future studies could always gather more information to continue making more ethical decisions. For example, surveys could be generated and trials could be run to determine if seniors with these conditions really might be confused by holograms while carrying out daily tasks. We can also study the impacts of remote caregiving, if that does become feasible given the advent of AR-driven care.

But most importantly, we see the ethical benefits of remembAR far outweighing the concerns, especially when considering quality of life improvements, socioeconomic gap tightening, and enabling of basic human liberties.

Challenges we ran into

Developing for Microsoft Hololens is incredibly difficult given limited documentation and preset settings. Instead, we designed a proof of concept using a phone camera.

Accomplishments that we're proud of

An absolutely beautiful end-to-end solution that is extremely well-presented in the front end and robust and efficient in the backend. The ability to create impact through both of the most important channels for patients of Alzheimers, dementia, and the like.

What we learned

Augmented Reality can be an extremely powerful tool if used correctly. Additionally, we learned that assistive technology is incredibly hard to design, given our lack of experiences with the actual condition. Finally, our biggest takeaway is an understanding of memory loss and its impact on families - there are countless articles detailing how difficult these lives can be.

What's next for remembAR

The most exciting part of remembAR is its modularity, flexibility, and scalability. It's modular in the sense that there are countless additions to make this a full-service virtual caregiver - task imitation, AR schedule annotator (e.g. project medicine schedule when recognizing a pillbox in the frame), or even dynamic captioning of persons.

Share this project: