Remi

Inspiration

We've all been impacted by the detrimental effects of neurodegenerative diseases eating away at the memories of our loved ones. The calls home are short, and when something happens like a fall, a scam call, or gradual signs of cognitive decline, the family is often the last to know.

That gap shows up in research too. A large English study linked dementia assessments to medical records. It found that fewer than half of people with dementia had a diagnosis on file in 2011 to 2013, and that people living with a spouse were more likely to be diagnosed than people living alone (https://quadram.ac.uk/publications/undiagnosed-dementia-in-primary-care-a-record-linkage-study/). Other research links slow changes in everyday speech, like longer pauses, slower talking, and more "um"s, to early cognitive decline. Those changes build gradually, so a family can easily miss them on a weekly phone call.

Conversation itself matters too. In the I-CONECT trial, adults over 75 who had frequent video chats with trained interviewers saw better cognitive scores if they had mild cognitive impairment. We wanted to offer some of that daily conversation.

We kept coming back to one thing our grandparents already do well: FaceTime. So we built someone they can call every day, who remembers their stories, keeps the family in the loop, and notices when something changes.

What it does

Remi is a FaceTime companion for older adults. There's no app to install and no test to take. They call Remi, and Remi:

Greets them by name and asks about their day. "Hi Hal, I'm Remi! How's your day been?" Then easy check-ins like "Did you get outside?" and "What did you have to eat?"
Remembers past calls, so it can pick up where they left off: "Last time you told me about the '78 blizzard…"
Brings back memories the family chose. Family members add a question, its answer, and a hint. If Grandpa is unsure, Remi offers the hints. It never says "wrong."
Shows family photos and asks the family's questions, crediting who asked: "Sam was wondering, what was your first car?"
Keeps the conversation engaging. Remi follows wherever the caller takes the conversation. It invites them to describe and explain ("How do you grow tomatoes that big? Walk me through it."), all so the call stays mentally active without ever feeling like a test.
Watches for red flags.
- Emergency (chest pain, trouble breathing, stroke signs, a fall they can't get up from): Remi tells them to hang up and call 911, and texts their emergency contact over iMessage.
- Thoughts of self-harm: Remi slows down, responds with care, gives 988 (and 911 if they're in danger), and texts the emergency contact.
- Urgent (a fall where they're okay, confusion about where they are, a stranger asking for gift cards or money): Remi texts the emergency contact so someone checks in today.
- Remi never says it called 911 or that help is on the way.
Notices slow changes in how they talk. Remi compares each call with that person's own usual. If a change lasts for weeks, it suggests mentioning it at the next check-up. Low mood is also accounted for.

On the family side, a web dashboard shows each call's summary, the fun facts, the photos Remi showed, retold stories, and how the caller sounded compared with their usual. It updates throughout calls.

Remi is not a medical device and doesn't diagnose anything. It helps families know when to check in.

How we built it

Calls. Chert provides an inbound FaceTime line that bridges into a LiveKit room. Our agent, built with LiveKit Agents (@livekit/agents) and OpenAI Realtime, talks with the caller in real time.

Conversation flow. A pure-JavaScript module builds Remi's instructions from the caller's context: greeting, day check-in, first-call consent, memory moments, the family's question, and wrap-up. Tools let the model log memory-moment outcomes and safety concerns without ever waiting on the network.

Safety. Concerns have three levels: watch (logged only), urgent, and emergency. Two triggers feed one escalation path. The first is the model's own report_safety_concern tool call. The second is a keyword backstop that scans the live transcript for first-person, present-tense phrases. Urgent and emergency concerns text the family through the Chert SDK's iMessage send, with idempotency keys so a retry never texts a family twice. If no contact is set, Remi never claims to have told anyone.

After the call. We transcribe the audio with word-level timestamps, then an LLM writes the summary and pulls out important information and memories.

Repeat memory and evaluation. The goal is to recognize when a caller brings back a story or question from an earlier call, even in different words, and link it to that earlier conversation. That gives Remi continuity and gives the family context for retellings. Retelling a favorite story can be intentional, not a sign of forgetfulness.

Build the detector. After each call, an AI extracts short descriptions of the caller's stories and questions. We turn each description into an embedding and compare it with extracted items from earlier calls. Matches above a fixed similarity threshold are saved with their earlier-call references. The results feed the call's repeat analysis. Prompting context is kept separate from whether the content is a repeat.
Create controlled test data. A reproducible script creates a fictional grandpa and four weeks of clearly labeled synthetic calls. 3 profiles of different scenarios are generated as well. The authored timing is a test fixture, not a measurement from real speech
Check the saved predictions against human review. A human reviewed all items. Of 30 repeated items, the detector caught 24 and missed six. It correctly left all 30 new items unflagged: no observed false alarms. All 24 flagged repeats linked to a correct earlier telling.

What this tells us. The detector can connect many differently worded retellings to earlier calls, and the six misses show where it needs improvement. These are preliminary results on a previously inspected synthetic corpus with one human reviewer. We kept the saved predictions and the 0.85 threshold fixed for this evaluation.

Speech markers. For each call, Remi measures the markers research points to:

Filled pauses ("um," "uh").
Pause length and frequency, which differ in early Alzheimer's.
Speaking rate, the most consistent speech measure.
Stories retold on different days, which stood out in phone conversations.
Vague words ("thing," "stuff") and vocabulary variety.

Comparing Grandpa with himself. A single conversation is only moderately reliable, while averages over several are reliable. So Remi never acts on one call.

Remi skips the first two calls, while the caller gets used to it, and builds a baseline from the next eight.
It scores each marker against that baseline and combines them into one weekly score. The less reliable markers, vague words and vocabulary variety, count half as much.
A note goes to the family only when the score stays high for 3 of the last 4 weeks and is still rising.
If mood has been low at the same time, the family gets a wellbeing card instead, since low mood can also change how someone talks.
A separate check catches a sudden drop in talking speed that persists over three calls.

Testing the rule. We simulated three people:

Early decline: speech shifts the way studies describe early cognitive decline. Pauses grow from about 650 to 850 milliseconds, speech slows, and fillers and retold stories increase.
Steady: speech stays the same, including through a week with a cold.
Low mood: speech is steady apart from two low-mood weeks.

Each person ran 200 times, about four calls a week with some calls missed. Noise for speaking rate and pauses comes from published reliability studies. Noise for the other markers is our own estimate.

Family dashboard. A Next.js app with SQLite, where family members add photos, questions, memory moments, and emergency contacts.

Consent and privacy. Sharing needs consent from both sides: the family proposes, and the caller confirms on a call. Nothing is shared until the caller agrees. We keep transcripts, summaries, and speech scores for analysis, but the family never sees transcripts. Data privacy in this domain is most important.

Challenges we ran into

Who speaks first. Our plan was to wait for the caller's "hello." Chert ends any call where the agent stays silent for 10 seconds after the line connects, so Remi had to greet first, timed to a precise signal signal.
"Call 911 immediately." We wanted Remi to dial 911 itself. Our line only takes incoming calls, software can't reliably place 911 calls, and automated calls to 911 are legally restricted. We redesigned around what we could guarantee: Remi tells the caller to call 911 and texts family right away.
False alarms from good stories. Reminiscence is full of past tense: "I fell in love at a church dance," "Dad had chest pains in '79." Our keyword backstop matches only first-person, present-tense phrases, and we wrote tests for stories it must not flag.
Crying wolf. Alerting on any unusual call would have messaged a steady family 74 times a year. Our first rule cut that to 1.5 a year, but it still mistook low-mood weeks for decline 22% of the time. That's why we added the 3-of-4-weeks rule and the mood check.
Spotting retold stories. Similar wording is not always the same incident, and a true retelling can use different words. In our 60-item human-reviewed synthetic evaluation, the detector caught 24 of 30 repeats at the fixed 0.85 threshold, with no false alarms among 30 new items. For each of the six misses, the correct earlier telling was the closest candidate but scored below the cutoff.
Memory without quizzing. Being quizzed can feel humiliating. We kept the family's memory questions but made "unsure" a normal outcome: hint, then story, never a correction.
Pauses. Older adults often pause mid-thought. An eager voice agent interrupts, which ruins the conversation and corrupts the pause markers, so we tuned turn-taking to wait longer.
Weeks of data in a weekend. Baselines take weeks of real calls, so the trends in our demo use clearly labeled simulated history. The calls, memory, safety alerts, and marker calculations all run live.

Accomplishments that we're proud of

A real FaceTime call to an AI that knows your family's questions, with no app on the caller's phone.
A safety path we'd trust for our own grandparents. It's tested to alert on "I fell and I can't get up" but not on "I fell in love," and Remi never claims help is coming when it isn't.
Memory moments that feel like a conversation, not a test.
A speech rule that doesn't cry wolf. In our simulations, Remi's rule:
- noticed 92% of changes that built up over about five weeks
- sent a steady family about one unneeded note every six years (0.16 a year)
- never mistook a low-mood stretch for decline

By comparison, alerting on single calls sent 74 unneeded notes a year. When the change unfolded over six months, Remi caught it in 99.5% of runs, typically about 11 weeks after it began, before the change was half done.

What we learned

Voice agents live or die on timing: when to speak, how long to wait, etc.
Never trust one call. Single conversations are noisy, so comparing someone with themselves over weeks is what makes a change worth mentioning.
Deciding when not to alert matters as much as the alert itself, for anything that messages a family.
Research first, thresholds second. We picked markers from published studies, then tested our thresholds in simulation before trusting them.
Read the platform's real constraints early. The 10-second rule and inbound-only calling reshaped our whole design.

What's next for Remi

-Medical research data gathered will help inform research on mental health diseases.

Real emergency dispatch through a certified provider, so an emergency alert can request help directly.
Messages and alerts in the family app. If Grandpa says "tell Sam I'm proud of him," Sam gets it. Safety alerts will also show on the family dashboard.
More languages, starting with Mandarin.
Smarter memory moments that bring back the questions people found hardest, spaced out over time.
A real-world pilot with families, with proper clinical and ethics review. We'd test the speech rule on real conversations, tune the thresholds with real data, and learn whether daily calls with Remi actually help.
Reaching more people, with regular phone calls or SMS for grandparents who don't have an iPhone.