Remember Me

Assistant for dementia patients

Comment

Inspiration

Patients affected with dementia are often in a state of confusion and repetitive questioning. When they forget things like whether they took their medication, verbal reassurance may not be enough be enough to convince them. Furthermore, dementia often turns once-familiar environments into obstacles. A microwave or a coffee maker can suddenly become an unrecognizable machine.

I built Remember Me to bridge these gaps. By turning "Did I take my meds?" into a moment of visual certainty with video proof and "How do I use this?" into an instant guided tutorial, I wanted to provide an assistant which can restore a sense of agency to the patient and help caregivers manage their cognitive load.

What it does

Remember the Past: Through the mobile interface, patients can ask questions like "Did I have my medication today?" If they did, the web interface displays the associated video clip as evidence to assure the patient.

"Show & Tell" Guidance: Through the mobile interface, a patient can point their camera at an appliance (e.g. microwave, coffee maker) and ask "How do I use this?" The system identifies the object (along with its brand and model) and automatically finds and plays a relevant instructional video on the main screen.

Understand the Present: Through the mobile interface, patients can ask questions like "What should I be doing right now?" if they feel disoriented. The system checks their pre-set daily routine in Firestore and provides a response to help them stay on track with their scheduled activities.

How it was built

Video Search: I integrated the Twelve Labs API (Marengo) to perform searches across camera footage using natural language.

The Backend: A Flask backend orchestrates a LangChain agent. It uses GPT-4o to decide when to query Firestore for the patient's routine or trigger a video search. If the user needs help using an appliance/tool, the backend leverages GPT-4o Vision to identify the tool from an image and utilizes Browser Use to find external tutorials.

Mobile Experience: Built with React Native (Expo), utilizing expo-camera for capturing an image of an appliance and OpenAI Whisper for converting the patient's speech to text.

Real-time Sync: I used Firestore as a real-time bridge to ensure that when the patient asks a question using their phone, the desktop device displays the response (e.g. video evidence) as well.

Challenges

One of the challenges was designing a user experience that wouldn't become an additional source of confusion for a patient already struggling with cognitive load.

I designed the mobile interface to be as simple as possible with just a couple buttons. The patient only needs to press a button and speak to the phone when they have a question which makes the application feel more like a natural conversation and less like a technical obstacle.

While the phone handles the input, I used the laptop as a larger screen for output. When a patient asks a question, they shouldn't have to squint at a tiny mobile screen. The web interface automatically triggers video clips or instructional tutorials, providing a clear view that is easy to see and process from across a room.

Lessons learned

I learned that in assistive tech, less is more because users need simple, intuitive UI and not a cluttered dashboard.

What's next for Remember Me

Fall Detection & Safety: Implementing real-time Twelve Labs monitoring to detect anomalies like falls or wandering and alerting caregivers.

Task verification: When a routine task reaches its expected completion time, automatically check recordings using Twelve Labs to see if the patient completed the task. If not, trigger a reminder for the patient or caregiver.

Built With

Share this project:

Updates