Inspiration
Dementia quietly steals continuity from everyday life: objects lose meaning, routines become confusing, and loved ones feel further away even when they are in the same room. We were inspired by caregivers who often act as a “human buffer” between the person and the world, constantly explaining, reminding, and emotionally supporting—often at the cost of their own wellbeing. At the same time, Gemini’s ability to see, hear, and speak in real time made us wonder: what if an AI agent could gently share that burden, not replace relationships, but stabilize them.
Dem.Dex Guide grew from a simple question: can a multimodal live agent help someone with dementia recognize their surroundings, remember who and what matters, and feel less alone. We wanted to create something that felt more like a calm, patient companion than a clinical tool, while still being practical enough for caregivers to trust in day-to-day use.
What it does
Dem.Dex Guide is a multimodal live agent that supports early-to-mid-stage dementia patients and their caregivers through three core capabilities.
- Live Companion: The user can speak naturally with Dem.Dex, which listens, responds with a warm voice, and stays aware of previous conversations, routines, and preferences throughout the day.
- Memory Bank: Using the camera, users or caregivers can point at familiar objects, rooms, or people; the agent uses Gemini’s vision to recognize and store labeled memories (“This is my bedroom”, “This is my daughter Anitha”).
- AI Art Therapy: When anxiety or confusion spikes, the user can ask Dem.Dex for calming scenes, such as “Show me the beach we used to visit,” and the system generates gentle, personalized imagery to anchor them emotionally.
In the background, Dem.Dex keeps a lightweight, privacy-aware daily diary of interactions and mood, along with a local Safety Profile that stores emergency contacts and key health notes for quick reference in moments of risk or disorientation.
How we built it
We built Dem.Dex Guide as a cloud-native live agent that combines Gemini’s multimodal capabilities with a robust backend on Google Cloud.
- Brain: We use Gemini via the Google GenAI SDK / Agent Development Kit to orchestrate a stateful live agent that can process voice, text, and camera input in real time, and respond with natural speech and visual outputs.
- Home: The backend is deployed on Google Cloud (for example, Cloud Run for the agent service, Firestore for structured profiles, and Cloud Storage for encrypted media and memory assets).
- Multimodal Pipeline: Audio is streamed into the live agent, which uses Gemini to transcribe, understand context, and generate responses; the vision path processes camera frames for “remember this” events and links them to the user’s Memory Bank.
- Safety & Diary: A lightweight orchestrator layer manages the Safety Profile checks (such as confusion signals or repeated emergency phrases) and appends summarized interaction logs into a health diary that caregivers can review.
The client UI is a simple, accessible interface optimized for tablets: large buttons, minimal navigation, and clear affordances for “Talk,” “Remember this,” and “Calm me.”
Challenges we ran into
- Balancing empathy and safety: Designing prompts and agent logic that are emotionally supportive without overstepping into clinical advice required careful phrasing, guardrails, and escalation patterns.
- Live agent robustness: Handling barge-in, partial speech, and abrupt topic changes while keeping the conversation coherent pushed us to iterate on the agent’s state management and error handling for timeouts and API failures.
- Vision in the real world: Lighting variations, cluttered rooms, and lookalike objects made it challenging to reliably tag and retrieve Memory Bank entries; we had to design flows where the caregiver can quickly correct or confirm what the agent “sees.”
- Privacy and trust: Because dementia care is extremely sensitive, we spent extra effort thinking through what should remain local, how to phrase consent, and how to keep caregivers in control without overwhelming them.
These constraints forced us to simplify aggressively and design for “reliably helpful” rather than “magically perfect.”
Accomplishments that we're proud of
- Moving beyond the chat box: We shipped a working prototype where the agent can see the scene, listen to the user, speak back, and capture memories visually—delivering the kind of multimodal UX that the challenge explicitly calls for.
- A calm, consistent persona: We designed and tuned an agent persona that feels slow, patient, and reassuring by default, which is very different from typical “fast and witty” assistants.
- Realistic caregiver workflows: The Memory Bank and Safety Profile are structured around how caregivers actually behave—short pockets of time, constant context switching, and the need to feel that the agent is additive rather than intrusive.
- Cloud-native architecture: We implemented a clear, testable architecture on Google Cloud with separation between live agent orchestration, user profiles, and media storage, aligning well with the judging emphasis on technical implementation.
Even at prototype stage, Dem.Dex already feels like something a caregiver could meaningfully try for a day.
What we learned
- Good multimodal UX is about pacing, not just features: Letting users talk, show, and listen is powerful, but the timing, confirmations, and repair strategies (for example, “Did you mean this photo?”) matter just as much.
- State and memory are everything: For dementia use cases, the difference between a generic chatbot and a useful companion is whether it remembers who you are, who you love, and what “home” looks like.
- Cloud and live agents require defensive design: We learned to treat latency, dropped connections, and partial inputs as the norm, building fallbacks and soft failures into the agent’s architecture.
- Designing for caregivers is a dual-user problem: Success means both the patient and the caregiver feel respected, in control, and emotionally safe; this deeply impacts copy, UI, and system behavior.
Overall, we came away with a stronger intuition for what “beyond text” actually feels like in a real-world, sensitive domain.
What's next for Dem.Dex Guide
- Smarter safety and escalation: Add optional integrations for caregiver alerts (such as SMS or push) when the agent detects repeated distress phrases, orientation issues, or broken routines, while still avoiding over-triggering.
- Richer personalization: Use long-term patterns from the diary (with consent) to adapt the companion’s tone, reminders, and art prompts to each user’s history, culture, and language.
- Multi-user homes: Support shared environments where multiple family members or caregivers interact with the same patient, with appropriate access controls and views.
- Clinical and research collaboration: Explore partnerships with dementia care organizations or researchers to evaluate whether Dem.Dex can measurably reduce caregiver burden, agitation episodes, or missed routines over time.
- Platform expansion: Package Dem.Dex as a deployable template on Google Cloud so other developers can extend it for related conditions (such as mild cognitive impairment or stroke recovery) using the same live agent foundation.
Log in or sign up for Devpost to join the conversation.