Recollect

An Alexa+ companion for people with dementia: it answers worried questions calmly every time, gives her memories back in her own words, and quietly alerts her family and doctor when something changes.

Comment

Inspiration

Dementia usually takes the newest memories first. Gurdev, the grandmother in our demo, can describe her wedding day in 1966: the bus to Patiala, the trunk on her lap. She cannot remember whether she took her blood-pressure tablet an hour ago. 57 million people live with dementia (WHO, 2021), and nearly 10 million more are diagnosed every year. Behind almost every one of them is a family.

Living with it hurts three people in different ways:

  • The person living with dementia spends the day in small, frightening uncertainties. Did I take my tablet? What day is it? Where is my grandson? She asks again and again, and she can feel people tiring of the answer. Being told "you already asked" or being corrected makes the anxiety worse, and late afternoons ("sundowning") and nights are often the hardest.
  • Her family and carers are tired, busy, or far away. Answering the same question for the twentieth time wears people down. A grandson in London cannot see how she is, and the small changes that matter go unnoticed until they become a crisis. Her stories, her life, start disappearing before anyone writes them down.
  • Her doctor sees her for ten minutes, a few times a year. The everyday conversations where change shows first never reach the consulting room. Research on speech in Alzheimer's disease shows that vocabulary, vague words, pauses and naming change gradually, but no one measures them at home.

We wanted something that lives where she already is, in her home and in her voice, and that makes life a little easier for all three of them. Alexa+ can now call outside tools through MCP, so we built Recollect.

What it does

Recollect is an Alexa+ companion built on one memory graph. That graph does three jobs.

1. For her: calm answers, every time

  • "Did I take my BP tablet?" → "Yes, you took your BP tablet at 8:10 today." The answer comes from the tablet log her family keeps. It is never guessed and never medical advice.
  • "Where is Aman?" asked three times gets the same gentle answer three times. Recollect never says "you already asked." The repeats are logged silently for the family.
  • "Who's this?" shows a who's-who card with a photo: "That's Aman, your grandson, Simran's son."
  • At dusk an evening calm routine turns on the lights, plays familiar music and keeps every reply to one short sentence. At night, if she wants to "go to the market", Recollect reassures her ("It's night time and everyone is asleep. You're safe at home.") and alerts the family if she keeps trying to go out.
  • She can speak English, Hindi, or a mix, and hears fixed, reviewed wording in her own language.

2. Her life story, in her own words

  • She says "I want to tell you a story" and talks. Every sentence is kept exactly as she said it, with the date.
  • When a memory slips away, she can ask "How did I come to Patiala?" and hear it back: You told me on 3 August 2026: "I came to Patiala in 1966, two days after my wedding. I was eighteen years old."
  • If she never told it, Recollect says so: "I don't have anything saved about that yet. Would you like to tell me about it?" It never invents a memory. No AI model writes an answer about her past.
  • Her family can explore her life on a live 3D life timeline. Each story sits at the year she said, or the year the family confirmed. Nothing is guessed.

3. For her family and her doctor: early, gentle notice of change

  • The family page shows her day at a glance: tablets taken and missed, stories told, and the questions she kept coming back to ("asked 3 times today: repeats are normal"). It also has her stories, a printable "This is me" profile for carers and hospitals, a carer check-in and a coach with "what to say" suggestions, and two-step consent with "delete everything".
  • Speech changes, measured only against herself. With consent from her and a family admin, everyday conversations become a natural speech sample. Recollect tracks vocabulary variety, vague words, pronouns, pauses, how often she names the people she loves, and how often she retells the same story. It compares them only with her own baseline.
  • A change is flagged only when it lasts four weeks on two or more markers, so one bad day never raises an alarm. The family then gets a gentle card ("Worth mentioning at the next check-up. This is not a medical assessment."), and her doctor gets a printable one-page report: weekly charts against her baseline band, questions worth asking, tablet adherence and how often she asked orientation questions.
  • It never diagnoses. A banned-word list ("diagnosis", "risk score", "signs of dementia"…) is tested over every sentence Recollect can produce.

How we built it

The MCP server (Alexa+)

A self-hosted MCP server (TypeScript, Streamable HTTP, protocol 2025-11-25) with 26 tools: 16 for her, 14 for her family, a few shared. Role tokens separate her tools from the family's. Five tools return MCP Apps cards (story, today, who's-who, speech trend, photo).

No model runs inside a tool. Every sentence she hears about memories, medicines or trends comes from fixed templates or from her own stored words. So every tool answers in milliseconds: slowest p95 144 ms measured against our AWS deployment, inside the Alexa+ 500 ms budget.

The memory graph: FalkorDB

Everything Recollect knows about a household lives in one FalkorDB graph, queried with Cypher. We mapped her memories like this:

(Session)-[:HAS_QUOTE]->(Quote {text, saidAt, private})      her exact sentence, with the date
(Quote)-[:MENTIONS {span}]->(Entity {kind, name, year})      the person, place, event or thing it is about
(Entity)-[:SAME_AS]->(Person)                               "my grandson" in a story = Aman in the family
(Entity)-[:RELATED {label}]->(Entity)                       her father -[brought]-> the radio
(Quote)-[:RETELLS {similarity}]->(Quote)                    the same story told again
(Person)-[:LOGGED]->(DoseEvent)-[:OF]->(Medicine)           today's tablets
(Person)-[:ASKED]->(AskEvent)                               the questions she asks
(Session)-[:MEASURED]->(MarkerSet)                          speech markers, never audio

Mapping memories as a graph instead of a pile of text is what makes Recollect trustworthy:

  • Finding the right memory. A question that names Patiala or her father starts at those entities and walks the MENTIONS edges to the sentences that mention them. Recall then scores each candidate:

$$\text{score}(q, s) = 0.6 \cdot \cos(\mathbf{e}_q, \mathbf{e}_s) + 0.4 \cdot \text{overlap}(\text{entities}_q, \text{entities}_{\text{story}(s)})$$

A local relevance classifier then checks the question twice, once as asked and once with the names hidden, so sharing a name with a story is never enough. The classifier can only choose among stored quotes or answer "nothing saved"; it never produces text.

  • Every link points at her real words. Each MENTIONS edge carries the span of text it came from, and the span must appear exactly in her quote. Entities that fail this check are dropped.
  • Noticing retold stories. A new story that closely matches an earlier one gets a RETELLS edge. That drives the retelling marker, and lets the family see what stayed and what changed between tellings. She is never told.
  • One source for every view. Her life timeline, the family page, the trend card and the doctor's report all read the same graph, so nothing is copied or re-summarised along the way.
  • Live. In the demo, a story told aloud reaches the graph within seconds, linked to her father, Patiala and the radio she has mentioned before.

Background agents on AWS

After she has been answered, background agents grow the graph:

  • Extract agent: a Strands Agents agent on Amazon Bedrock (Amazon Nova Pro, structured output, temperature 0), hosted on Bedrock AgentCore Runtime, proposes the people, places, events and things in each sentence. A deterministic validator decides what reaches the graph.
  • Markers agent: computes text, graph and (with consent) timing markers per session, and deletes any audio after analysis.
  • Trends agent: builds her baseline and flags sustained change. For each metric $m$ it uses the baseline median $\tilde{x}m$ and a robust spread $\sigma_m = 1.4826 \cdot \text{MAD}_m$, and flags a weekly value $x{m,w}$ only when

$$\frac{x_{m,w} - \tilde{x}_m}{\sigma_m} \ge 1.5 \text{ (in the worse direction) for 4 consecutive weeks,}$$

on at least two metrics before any family card appears. Vocabulary variety uses the moving-average type–token ratio, $\text{MATTR} = \frac{1}{N-W+1}\sum_{i=1}^{N-W+1} \frac{\text{types}(w_i \dots w_{i+W-1})}{W}$ with $W = 50$.

The Alexa+ simulator and family page

  • Alexa+ simulator (React): a real MCP client. Amazon Bedrock Converse picks the tool, and the simulator speaks the tool's answer verbatim, discarding the model's own text. It has push-to-talk, story mode, MCP Apps cards, a latency strip, a simulated home (lights, speaker) and a live memory-graph panel.
  • Family page (React) with the doctor report, the "This is me" profile and the 3D life timeline (three.js).
  • Agent Skill (skill/SKILL.md) that teaches any assistant how to support someone with memory loss using these tools.

Evidence

Check Result
Grounding eval (30 questions: 20 answerable, 10 not) 30/30, 0 invented answers
Held-out questions written after tuning 10/10, 0 invented
Tests 133 TypeScript + 51 Python, all passing
Latency on AWS (50 calls per tool) slowest p95 144 ms (budget 500 ms)

Challenges we ran into

  • Never inventing a memory. Matching on names alone returned real but wrong quotes: "What was my mother's name?" got a sentence about her mother's cooking. The fix was to check every question twice, once with the names blanked out, so a shared name is never enough. "Nothing saved" turned out to be one of the most important answers Recollect gives.
  • A safety bug that only appeared in the evening. In our first live run at 17:34, "What day is it?" ended with "your next tablet is at 8:00", a dose missed nine hours earlier. For someone with memory loss that invites a double dose. Now only a dose that is upcoming or at most an hour late is ever offered as "next"; missed doses go to the family page instead.
  • Speed. Running two local inferences at once pushed recall from under 300 ms to 4.6 s, and a new AgentCore session per call cost 11–12 s per sentence. We serialised inference and reused AgentCore sessions; a story now reaches the graph in about 8 s.
  • Honest change detection. Weekly speech samples are small and noisy. We flag only sustained, multi-marker change against her own baseline, and test that a bad week, a single noisy marker or a recovered dip never raises a card.
  • Hindi in tool calls. Some Hindi sentences broke tool use on Nova. We now translate her question to one English line before the tool loop, but pass her stories exactly as she said them, never translated.

What we learned

In a health-adjacent assistant, most of the safety comes from what the model is not allowed to do. Models choose tools and propose graph links; fixed wording and her own words do the talking. We also learned that the gentlest design is often a refusal to act on something: never correcting her, never mentioning a repeat, and never diagnosing. A graph made those rules enforceable, because every answer can be traced back to a sentence she actually said.

What's next

A pilot with families and carers (with ethics review), consented on-device acoustic features if Alexa+ offers them, more languages (Punjabi first), and speech markers validated per language.

All people in this project are fictional. The 12-week speech history is simulated and labelled so on screen. Recollect is not a medical device and makes no diagnosis.

Built With

Share this project:

Updates

Submission history