Inspiration
57 million people live with dementia today. By 2050 that number reaches 153 million, and someone new is diagnosed about every 3 seconds (WHO; The Lancet Public Health, 2022).
Picture Mary, 78. Every Sunday her daughter Sarah FaceTimes her. Sarah asks, "What did you do today, Mom?" Mary can't remember, and some weeks she isn't sure who's calling. The call ends in silence. The WHO lists not recognizing family as a symptom as dementia progresses.
In the Odyssey, Calypso kept Odysseus on her island and made him forget home. We built the inverse: Calypso⁻¹, which brings people home to their memories.
What it does
Calypso⁻¹ joins Mary's FaceTime calls as a quiet guest. When a relative's face appears, Mary sees a card on Calypso⁻¹'s video tile: "Sarah, your daughter." When someone asks "What did you do today?", she sees her own day and can answer for herself. Mary doesn't learn a new app. She just answers FaceTime.
- The puck: a Raspberry Pi with a lav mic that Mary wears. It records her day, works with no Wi-Fi, and uploads once it reconnects.
- One-time setup call: each relative FaceTimes Calypso⁻¹ once. It asks their name and how they know Mary, then saves their face and voice.
- Identity cards in group calls: in a FaceTime with Mary and a relative, Calypso⁻¹ matches faces and voices and shows who's there, one card at a time.
- In-call recall: Claude Haiku spots questions about Mary's day, and Calypso⁻¹ answers with a card built from what the puck heard.
- iPhone app and judge dashboard: the SwiftUI app handles onboarding, people, and Mary's Today timeline. A live dashboard shows every decision: puck audio, voice match, face score, and recall.
It never guesses. Below the confidence bar, the card says "Getting to know someone new…" instead of naming the wrong daughter. Places work the same way: "somewhere with café sounds," never a made-up name.
How we built it
The flow:
- Puck → Vultr: the puck spools audio to disk and uploads it to our FastAPI backend on Vultr.
- Vultr backend: Grok transcribes, Claude summarizes, and SQLite + Redis/RQ store and process everything.
- FaceTime → Chert → LiveKit: Chert bridges the call into a LiveKit room, where our Calypso⁻¹ agent (also on Vultr) sees and hears it.
- Agent → FaceTime: the agent asks the backend who's there and sends cards back as its outgoing video.
- iPhone app + dashboard: the SwiftUI app syncs with the backend, and an event bus streams every decision to the judge dashboard over SSE.
Every part talks to the others only through the backend's HTTP API, with one shared pydantic contract.
- Hardware / edge: Raspberry Pi firmware in C++ (low-latency mic capture + on-device voice ID), Google Coral Edge TPU, Saramonic lav mic, NVIDIA NeMo TitaNet, Bluetooth LE to the phone, 3D-printed mount
- Backend: Python, FastAPI, SQLite, Redis + RQ job queue, server-sent events
- AI/ML: Claude Sonnet (setup calls, daily summaries), Claude Haiku (detecting recall questions), xAI Grok (speech-to-text and text-to-speech), OpenCV YuNet + SFace (faces), TitaNet (voices)
- Calls: Chert, LiveKit Agents, Pillow (renders cards into the agent's video)
- Frontend: SwiftUI (iOS 17+), plain HTML/JS dashboard
- Infrastructure: Vultr, systemd
Every provider (STT, TTS, face model, voice model, call topology) sits behind one interface with a fake version, so swapping one is one env var and the tests never need live keys.
How we used Vultr
Vultr runs everything in the call path. One Vultr server hosts three systemd services that start at boot and restart on failure:
calypso-backend: the FastAPI API, SQLite database, Redis + RQ workers, face and voice models, and all Claude decisionscalypso-media: the LiveKit media worker that sees and hears the call and sends cards back as videocalypso-chert: the Chert FaceTime connector
No laptop sits in the call path. The puck and the iPhone upload straight to Vultr, the agent calls the backend over loopback, and secrets stay in root-only env files on the server.
How we used hardware
The puck is Mary's memory. It's a Raspberry Pi with a Google Coral Edge TPU, a Saramonic lav mic, a power bank, and a 3D-printed top plate. The puck firmware is written in C++ so mic capture and on-device voice ID stay low-latency on the Pi.
- Offline first: every audio segment is written to an on-disk spool before anything else. When the network returns, it uploads oldest first. Each segment gets a UUID at capture, so retries never create duplicates on the server.
- Clock-safe: if the Pi boots without network time, segments are held and their timestamps are corrected once the clock syncs, so Mary's day stays in the right order.
- Coral Edge TPU as the offline brain: when Wi-Fi drops or the signal is gone entirely, the puck does not wait for the cloud. It flips into an offline path and runs voice ID on the Coral Edge TPU attached to the Pi, not on the Pi's CPU. We compiled NVIDIA NeMo TitaNet into an Edge-TPU TFLite model so speaker embeddings stay on the stick. While the backend is reachable, the puck syncs a local gallery of voiceprints; when the link dies, that gallery plus the Coral is enough to keep answering "who is talking" with no internet.
- Output even when the cloud is unreachable: identity results still leave the puck. Over a custom Bluetooth LE service, the Coral's "who is talking" output streams to Mary's iPhone, so the app keeps getting live speaker labels during a total outage. The same BLE path is used during onboarding to teach the puck Mary's voice. Audio keeps spooling to disk in parallel, so nothing is lost for later upload once signal returns.
- Wearable build: lav mic, power bank, and a 3D-printed top plate so the whole stack sits on Mary as one device, not a pile of adapters.
How we used Chert
Chert is what makes FaceTime possible. It bridges a real FaceTime call into a LiveKit room, so our Python agent can see and hear the call like any participant.
- Setup calls: a relative FaceTimes the Calypso⁻¹ contact. The agent talks with them (Grok voice, Claude for the conversation), and the backend saves their name, relationship, face prints, and voice prints, tied to their verified caller number.
- Cards as video: we render each card with Pillow and publish it as the agent's outgoing video, so the cards show up on Calypso⁻¹'s FaceTime tile. Nothing gets installed on Mary's phone.
- Automatic routing: each incoming call is routed to setup or group mode by who is calling.
- Silent in group calls: the agent speaks only to confirm a new person. Otherwise it shows one card at a time, for 6 to 10 seconds.
How we used SpaceX AI (xAI Grok)
Grok is Calypso⁻¹'s ears and voice.
- Speech-to-text: Grok transcribes the puck's audio on the backend. Those transcripts feed Claude Sonnet, which builds Mary's day.
- Live call speech: in setup calls, Grok transcribes the relative and speaks Calypso⁻¹'s replies through the LiveKit media worker.
Both sit behind our STT and TTS interfaces (CALYPSO_STT, CALYPSO_TTS).
Challenges we ran into
- Getting into a FaceTime call at all. Chert's LiveKit keys weren't provisioned at first (
livekit_not_provisioned), so we brought our own LiveKit project. We also hit worker sessions stuck "cleaning up" and calls missing from call records, and had to tell provider problems apart from our own bugs. - Group calls. Automatic group dialing and native hangup weren't available through the CLI, so the agent asks the caller to hang up and the group call is started from FaceTime.
- Not showing the wrong person. Reviewing our deployed build, we found that a demo test setting could override a real face match. We made it opt-in and off by default. We also found that two relatives sharing one phone number could merge into one profile, which is still on our list.
- Audio that survives bad Wi-Fi. Fixed 3-second chunks cut words in half and wasted uploads. Making the queue durable, idempotent, and clock-safe on a Pi took more care than recording did.
- Edge compute that still speaks when the radio dies. Compiling TitaNet for the Coral, keeping a local voice gallery in sync, and shipping identity over BLE while the upload spool holds audio meant the puck had to stay useful with zero cloud — not just buffer files and wait.
- Four parallel tracks in 24 hours. Puck, backend, agent, and iOS were built by four people at once. A single shared data contract and fake versions of every provider let each of us build before the others were done.
Accomplishments that we're proud of
- Real FaceTime enrollment works end to end. A relative calls Calypso⁻¹, has a short conversation, and their name, relationship, face, and voice are saved on our Vultr backend.
- Mary doesn't install anything. The cards are Calypso⁻¹'s own video in an ordinary FaceTime call, the only interface many people with dementia already know.
- Safety is built in, not added on. Face matches need a minimum score and a clear lead over the next-closest person, or the card stays neutral. Photos flagged as sensitive are never shown.
- A puck that keeps working offline. It records to disk, uploads later without duplicates, and when the signal is gone it still runs voice ID on the Coral Edge TPU and pushes "who is talking" to the phone over Bluetooth LE.
What we learned
- How to bridge a consumer app like FaceTime into a programmable media pipeline with Chert and LiveKit, and how to draw on an agent's outgoing video.
- How to design for a user who can't tell you the system is wrong: confidence thresholds, margins between candidates, and neutral fallback wording.
- How to make edge uploads reliable: write to disk first, fix IDs at capture, retry with backoff, and correct clocks later.
- How to treat a Coral Edge TPU as a real fallback path: sync a gallery while online, run embeddings on the stick when the link dies, and keep delivering results over BLE instead of waiting for the cloud.
- How to split work between AI models: Haiku for fast decisions, Sonnet for writing, Grok for speech.
- How to let four people build in parallel against one API contract with fake providers.
What's next for Calypso⁻¹
- Automatic group calls. Have Calypso⁻¹ dial Mary and a relative together, and let Mary say "call my daughter."
- iMessage intake. Let relatives text photos and memory notes that feed Mary's recall.
- Tune on real families. Pilot with memory-care families and tune face and voice thresholds on real lighting, cameras, and voices.
- A caregiver view, and consent. Build a caregiver view and a clear consent flow for the people Mary's puck hears. Calypso⁻¹ is a communication aid, not a medical device, and we won't make medical claims without evidence.

