Inspiration
Margaret is 78, has early memory problems and lives alone. Her daughter Lisa is 400 miles away. Millions of families are in the same position: the parent wants to stay home, and the child wants to know she is safe without hovering. Memory decline is also what makes someone a scam target, so we wanted one device that looks after her physical, mental and financial health together.
What it does
Pip runs on a Vexo Open wristband: a motion sensor and a microphone, nothing to press and no screen.
- Physical. Motion from the band feeds fall detection, steps and walking regularity. A fall brings up a "did you fall?" check that Margaret can answer by voice or by tapping the band, and Lisa is alerted.
- Mental. The microphone is always on. Everything said near her wrist is written down as it is spoken, attributed to each voice, and stored as a searchable memory of her day. She can ask out loud, "Hey Pip, did I take my pills this morning?", and Pip answers from her own day. Each night the day is condensed into a summary for Lisa and notes for her doctor.
- Financial. When a conversation turns into a request for gift cards, account details or secrecy, her spending card freezes on the spot and Lisa is told. Only Lisa can unfreeze it. When Margaret mentions a need during the day (her knee on the stairs), the nightly summary suggests HSA/FSA-eligible products Lisa can buy through Stripe.
How we built it
The band connects straight to a web page over Bluetooth using the Vexo Open kit. Motion is analysed in the page. Microphone packets stream to a small Python server, which sends them to Deepgram for live transcription and runs a local speaker model (WeSpeaker CAM++) on each phrase to work out whose voice it is. Phrases go into SQLite and are searched by words (BM25) and by meaning (vectors), fused by reciprocal rank and filterable by person. Google's Gemini decision model watches the last 45 seconds of conversation for scams and calls for help. Gemini answers Margaret's spoken questions from the search results and writes the nightly summary, Deepgram's Aura voice reads answers aloud, and Stripe Checkout handles purchases.
Challenges we ran into
- Through a wrist microphone, 11% of words were wrong, and the scam line a keyword list would rely on ("your grandson is in jail and needs bail money") came out as nonsense.
- Voice identity behaves very differently on a wrist microphone than on clean recordings. Our first thresholds labelled the wearer's own speech "not sure", and a one-second "I'm fine" is too short to identify anyone.
- Pip heard its own voice through the microphone and interrupted itself. We taught it to recognise its own words instead of going deaf while speaking, so Margaret can still interrupt it.
- Stripe Issuing never opened in our sandbox, so the limited spending card is simulated in our code. Stripe Checkout is real, in test mode.
Accomplishments that we're proud of
- The whole chain works live on the real band: speech to transcript to voice label to scam alert to frozen card, in a few seconds.
- Scam detection works from Margaret's side of the call alone, because in real life the band often cannot hear the caller.
- We measured instead of assuming, and published the numbers in
FINDINGS.md. - We say plainly what is real and what is simulated.
What we learned
- Scams must be judged over a stretch of talk. Sentence by sentence, three of seven fragments looked innocent; judged together, the model called it a scam at 98%.
- A model reading only the transcript is blind after a fall: a groan produces no text. The same model listening to the audio scored a real groan at 87% for distress and a laugh at 11%.
- A model must never be the thing that cancels an alert. In testing it confidently cancelled a fall alert because a TV weather forecast "answered". Our rule: it can raise an alert or ask again; only a person, or a tap on the band, cancels.
What's next for Pip
Distress detection from audio in the live fall check; the band on a belt or ankle for gait asymmetry and balance, where the published methods put the sensor; real limited cards through Stripe Issuing; and consent and privacy controls for everyone the microphone hears. Pip is a hackathon project, not a medical device.
