ReMindAR

Inspiration

The inspiration for developing augmented reality (AR) glasses for dementia patients came from the recognition of a growing global challenge: the impact of dementia on individuals and their caregivers. Dementia is a progressive condition that affects memory, cognition, and the ability to perform everyday tasks. Millions of people around the world struggle with the effects of dementia, which can lead to a loss of independence and a diminishing quality of life. Many existing solutions, such as memory aids or caregiver assistance, either fall short in terms of scalability or place additional financial and emotional strain on families. This reality pushed us to think about how technology could step in as a supportive tool to enhance the lives of dementia patients in a meaningful, affordable way. Our vision was to create something that wasn’t just another piece of assistive technology, but an integrated companion that could help users navigate their day-to-day lives with greater ease and dignity. We wanted to make a device that didn’t just remind people about their medication or recognize faces but could actively help patients with things like remembering where they left their belongings or summarizing their day so they can recall important moments. Inspiration also came from observing the frustration and confusion dementia patients face, such as struggling to recognize loved ones or feeling disoriented in familiar places. We saw the potential for AR to fill these gaps, giving patients a tool that could keep them more connected to their environment and to the people around them. Moreover, we were driven by the idea of making advanced technology accessible. Current solutions in the market are often bulky, expensive, and require significant effort to set up, which limits their usability. We wanted to change this by utilizing open-source hardware and leveraging affordable but powerful software solutions to create something that was both practical and innovative. Our ultimate goal was to design AR glasses that could not only make everyday tasks easier for dementia patients but also empower them to live independently, while providing peace of mind to caregivers. This deeply human-centered approach is what fuels our desire to continually improve and expand the functionality of the product.

What it does

The augmented reality (AR) glasses for dementia patients are designed to serve as a comprehensive, real-time support tool, enhancing the user's ability to manage daily tasks, stay connected with loved ones, and maintain a sense of independence. Each feature of the glasses is carefully designed to address common challenges faced by dementia patients, using advanced technology to assist them in ways that simplify their lives without feeling invasive. One of the key functions is the medication reminder system, powered by computer vision. Dementia patients often struggle to remember when and which medications to take, leading to missed doses or confusion about their treatment plan. The glasses can automatically recognize different medications and provide reminders at appropriate times, ensuring the patient adheres to their prescribed routine. This not only improves health outcomes but also relieves some of the caregiving burden associated with medication management. The glasses also feature facial recognition technology to help patients identify family members and friends. For individuals with dementia, recognizing familiar faces can become a daily struggle, causing stress and emotional strain. By using AI-powered facial recognition, the glasses display the name and relationship of the person in view, helping the patient feel more at ease and socially connected. This feature also reduces anxiety, which often arises from memory lapses, and helps maintain important personal relationships. To support daily organization, the glasses track activities using live and recorded footage, helping patients stick to schedules and complete their tasks. For example, if the patient is supposed to engage in an exercise routine or attend an appointment, the glasses provide reminders and track their progress. Additionally, the system can capture the patient's day, summarizing activities and creating an automatic journal. This summary helps them reflect on their day, strengthens memory retention, and provides caregivers or doctors with a clear view of the patient’s routine, facilitating better care management. Another critical feature is the easy access to emergency contacts and important information such as home address, ensuring that in moments of confusion or distress, the patient can quickly access help. This reduces the risk of getting lost or facing dangerous situations without assistance. Furthermore, the glasses assist with an everyday problem: forgetting where things are. The object location memory feature tracks the location of important items like keys, wallets, or glasses, offering real-time reminders about where the patient left these objects. This drastically reduces the frustration that comes with misplacing items, a common issue for dementia patients, and helps maintain a sense of control over their environment. Lastly, the system’s real-time contextual information provides useful insights about the world around the patient, such as location, helping them orient themselves in familiar or unfamiliar surroundings. All these features combined allow dementia patients to regain a sense of autonomy, navigate their day with greater ease, and reduce the emotional and mental toll that dementia takes on them and their caregivers.

How we built it

The development of these augmented reality (AR) glasses for dementia patients was rooted in a combination of open-source hardware and software solutions, designed to keep the product affordable, lightweight, and highly functional. Our goal was to create a device that could seamlessly integrate into the daily lives of dementia patients without the bulkiness or high cost of existing solutions like virtual reality (VR) headsets. To achieve this, we carefully selected hardware components that would allow us to strike the perfect balance between power and portability. This ensured the glasses could be worn comfortably all day, just like a normal pair of glasses, rather than requiring heavy, cumbersome gear. We built the system primarily using Python, which provided the flexibility needed to integrate various technologies efficiently. One of the core technologies leveraged was Bluetooth Low Energy (BLE), which enabled low-power communication between the glasses and other devices. BLE technology allows the glasses to remain connected to external systems—such as smartphones or other wearable devices—without draining battery life, a critical feature given the need for continuous use throughout the day. This ensures the glasses can function independently and maintain communication with external data sources without the need for constant charging. For software, we integrated advanced computer vision algorithms to enable the real-time recognition of medications and faces. This required careful optimization to ensure that the glasses could identify pills and familiar faces even in varying lighting conditions or angles. Using Kindo AI and Twelve Labs APIs, we incorporated the ability to vectorize and summarize video footage, which allows the glasses to track the patient's activities and create an end-of-day summary. This functionality helps the patient maintain a daily journal and provides caregivers with valuable insights into the patient’s routine. The APIs also enable the glasses to interpret video data efficiently, even with limited processing power, helping us maintain the balance between performance and hardware limitations. In terms of hardware, we relied on open-source components to make the glasses cost-effective while still delivering high-performance features. This also allowed for customization, ensuring the final product was more compact and user-friendly compared to other market alternatives. By using open-source hardware, we avoided the typical trade-off between affordability and functionality, resulting in a solution that doesn’t compromise on quality while remaining accessible to a wider audience. Designing a wearable AR system that could process live data without being overly bulky was one of the main challenges we faced. However, by keeping the processing requirements low and offloading more demanding tasks to external systems via BLE and cloud processing, we were able to ensure the glasses remain lightweight and practical. The combination of efficient software and minimalistic hardware resulted in a device that can be worn comfortably, while still providing all the advanced functionality required to assist dementia patients in their daily lives.

Challenges we ran into

One of the biggest challenges we faced was balancing processing power and portability. Since the glasses needed to perform complex tasks like real-time facial recognition, medication identification, and video summarization, we had to optimize the system to ensure it could handle these demands without becoming bulky or overheating. Additionally, ensuring a seamless user experience with limited hardware resources required careful optimization of both the computer vision algorithms and BLE communication to avoid draining battery life. Another hurdle was integrating all the features in a way that would make the glasses easy to wear and intuitive for patients, who often need a simple, user-friendly interface. Managing all these elements while keeping the glasses affordable and lightweight required significant iteration and refinement.

Accomplishments that we're proud of

Successfully integrating AI-driven features such as facial recognition and video summarization on open-source hardware, enabling patients to identify loved ones and track daily activities. This feature offers independence to patients while providing valuable insights for caregivers. Another accomplishment is leveraging BLE technology, ensuring the glasses are energy-efficient, portable, and able to support real-time functions like medication reminders and object tracking throughout the day. Lastly, we’re proud of the user-friendly design, offering patients a sleek, comfortable, and intuitive device that empowers them to live more independently.

What we learned

Throughout this project, we learned the importance of balancing advanced technology with user simplicity. Dementia patients need intuitive solutions, and while integrating AI and AR can offer incredible support, the user interface must remain straightforward. We also discovered how crucial it is to optimize performance on lightweight, portable hardware without compromising functionality. Additionally, talking to mentors and peers who have experienced this through a family member in the past taught us that adaptability and comfort are as important as technical innovation in creating a product that can truly improve the quality of life for dementia patients.

What's next for ReMindAR

Our immediate focus is on enhancing the resolution of the head-up display, transitioning from monochrome to full RGB, which will offer a sharper, more visually engaging experience for users. In addition, we plan to make the glasses completely wireless, eliminating the need for physical connections and increasing both mobility and comfort, making the device feel even more like a regular pair of glasses. Shifting the processing to the cloud is another key development that will allow for more powerful, real-time functionalities without straining the hardware, improving battery life and performance. These improvements will make the glasses more user-friendly and scalable, positioning them as an even more versatile and impactful tool for dementia care. We also aim to continue refining the AI and AR capabilities to further assist patients in their daily routines, enhancing both their independence and quality of life.

Built With

• ble
• kindo
• python
• twelvelabs