-
-
memento- Ai companion
Inspiration
A few weeks ago, my neighbor walked out of her house because of dementia. Her family found her an hour later, standing on a street she'd known her whole life — completely lost.
It brought everything back. My grandfather had Alzheimer's. The hardest part wasn't the disease itself. It was knowing he was sitting alone in the evenings, confused, wondering if anyone was there — and we couldn't always be.
There was nothing patient enough to sit with him when we couldn't. Nothing that could answer the same question for the hundredth time without making him feel like a burden. Nothing that could call us the moment something felt wrong.
We built Memento because we wish it had existed for him.
What It Does
Memento is an ambient, always-on AI companion for people living with dementia. It runs on a screen in the patient's home and uses Gemini's Live API to see, hear, and respond in real time — like a trusted presence that never leaves the room.
Live Conversation Warm, natural voice conversations 24/7. Memento answers repeated questions without frustration, gently grounds the patient when confused, and maintains a calm familiar presence throughout the day.
Visual Orientation Memento proactively identifies objects the patient picks up — medication bottles, appliances, photos — and offers real-time guidance without being asked. No button to press. No command to give.
Face Recognition When a family member enters the room, Memento recognizes them from the Memory Bank and greets them by name — sharing something warm and specific about the patient's day so every visit feels connected.
Memory Book Caregivers upload cherished family photos, voice notes, and life stories. Memento draws from this library to comfort and ground the patient during moments of loneliness or confusion.
Cinematic Memory Generation When a patient expresses longing for something not in their Memory Book — "I wish I could see my old garden..." — Memento uses Imagen 3 to generate a vivid personalized image and Veo 3.1 to bring it to life as a cinematic memory, narrated warmly in real time.
Caregiver Safety Layer Memento monitors continuously for signs of distress or physical danger. If something concerning is detected, it silently dispatches a real-time SMS alert to the care team — while keeping the patient completely calm.
Daily Interaction Summary Every evening, caregivers receive a synthesized summary of the patient's mood, what they talked about, what they remembered, and any concerns — so families stay close even from far away.
How We Built It
Memento is built on three Gemini-native layers working in absolute concert. The architecture is a Node.js WebSocket backend communicating with a Next.js frontend, heavily relying on Agentic Function Calling.
Perception & Conversation: The Gemini Live API (Multimodal) serves as the core engine. It processes live audio and video simultaneously, maintaining context with near-zero latency. Memory & Identity: A structured "Memory Bank" configured during onboarding stores the patient's family relationships, cherished memories, medication schedules, and anxiety triggers. This is injected into every session. Generation & Action: We utilize Gemini Tool Calling to drive Background Execution. Imagen 3 generates on-demand visual memories, Veo 3.1 creates cinematic animations, and Twilio dispatches real-time alerts. To determine when Memento should trigger an emergency alert without false positives, we developed a weighted threshold algorithm optimized in the system prompt. The decision to call a caregiver $C_{alert}$ is defined as a function of Anxiety $A(t)$, recognized Environment Hazards $H$, and Time of Day variance $\Delta T$:
$$ C_{alert} = \begin{cases} 1, & \text{if } \alpha \int_{t-5}^{t} A(t)dt + \beta \sum H_i + \gamma \Delta T > \theta_{critical} \ 0, & \text{otherwise} \end{cases} $$
By fine-tuning these scalar weights ($\alpha, \beta, \gamma$), the model reliably differentiates between normal momentary confusion and a true wandering risk requiring intervention.
Challenges We Ran Into
Silence during tool calls When Memento triggered a tool call, the model would go quiet for several seconds. For a dementia patient, unexpected silence is disorienting and frightening. We solved this through a conversational continuity protocol — Memento speaks warmly throughout every tool execution so the patient experiences no gap, only a seamless moment where a memory simply appears.
The repetition problem Standard LLM behavior includes subtle signals of impatience when handling repeated questions. For dementia patients, this is genuinely harmful. We built an explicit repetition protocol ensuring Memento treats every question as if it's the very first time — every single time.
Face recognition confidence An incorrect name during face recognition causes real distress. We implemented a strict confidence threshold — Memento only greets by name on a high-confidence match, and defaults to a warm open-ended greeting when uncertain. Getting it wrong once is one time too many.
Proactivity without intrusiveness Memento needed to notice a medication bottle being picked up unprompted — but not feel overwhelming or intrusive. Finding the right cadence of when to speak versus when to observe quietly required extensive prompt engineering and behavioral testing.
Dispatching alerts without alarming the patient When Memento detects a safety concern it must alert the caregiver immediately — but cannot let the patient sense anything is wrong. We engineered the system prompt so Memento simultaneously dispatches the alert AND speaks more slowly and warmly than usual. The patient feels nothing but calm.
Accomplishments That We're Proud Of
We're proud that Memento actually feels human. It doesn't rush. It doesn't forget to be kind. It notices things. It remembers what you said five minutes ago and brings it back at exactly the right moment.
We're incredibly proud of the cinematic memory generation pipeline. Watching a patient say "I wish I could see my old garden" and seeing a beautiful, personalized image appear within seconds via Imagen 3—brought to life with warm narration—is the kind of moment that proves technology can do profound good.
Most of all, we're proud that we built something we wish had existed for our own families.
What We Learned
The hardest problems in AI aren't just technical—they're deeply human.
Getting a model to generate an image is easy. Getting it to hold a confused, frightened person's hand with exactly the right words, at exactly the right moment, in exactly the right tone—that takes relentless iteration.
We learned that prompt engineering is product design. Every behavioral nuance—the warmth, the pacing, the silence rules, the repetition handling—lives in the system prompt. Change one line, and you change the entire soul of the companion.
We also learned that families of dementia patients carry an invisible weight every single day. The anxiety of not knowing. The guilt of not being there. Memento doesn't just serve the patient—it gives the family their life back.
What's Next for Memento
Voice Personalization Training Memento to speak in a voice that sounds familiar — a family member's recorded voice — so the patient hears someone they love, not a stranger.
Long-Term Memory Across Sessions Memento currently operates within a single session. The next step is persistent memory — so it remembers that yesterday she talked about Maine, and brings it up naturally today.
Caregiver Mobile App A dedicated app for real-time alerts, the daily summary, Memory Bank management, and mood trend tracking over weeks and months.
Clinical Partnerships Working with dementia care facilities and neurologists to validate Memento's impact on patient anxiety, caregiver burden, and quality of life — and to ensure it meets the safety standards required for medical-adjacent deployment.
Multilingual Support Dementia doesn't speak one language. Expanding Memento to support regional languages and dialects so it can serve families everywhere.
Memento doesn't replace family. It makes sure their loved one never has to sit alone in the silence.
Built With
- gcs
- googlecloudbuild
- googlecloudrun
- googlegeminiliveapi
- googleimagen3
- lucide
- nextjs
- node.js
- tailwindcss
- twilioprogrammablevoice
- veo3
- websockets
Log in or sign up for Devpost to join the conversation.