AR View Outside
Landing
People Context Management
People Facts
Visitor Input
Consent
Face Input

Inspiration

One of us watched both grandparents live with dementia, and both eventually stopped recognizing him. He'd walk into the room and feel like a stranger to them. They knew he was someone, just not who. It hurt him, and it seemed to hurt them too, being surrounded by people who clearly loved them without knowing why.

Another teammate's mother works at a nursing home and cares for residents with dementia every day. She has told her son about the sadness she sees there: people losing the faces and names of the ones they love. Dementia doesn't take a memory from one person. It takes a relationship from everyone in it.

We couldn't give anyone their memories back. But even a small hint, like "This is your grandson," could let those visits start with warmth instead of confusion.

These stories aren't rare. WHO notes that people with dementia may not be able to recognize family members or friends. One Dutch nursing-home account (Verbeek et al. 2020) describes a man who hadn't seen his son for 10 weeks, crying because he "could not understand how he could forget about his son."

Some stats to look at:

7.4 million Americans 65+ live with Alzheimer's dementia in 2026; 57 million people worldwide lived with dementia in 2021.
12.7 million unpaid caregivers gave 19.6 billion hours of care in 2025, 472,000 of them in Pennsylvania.
39–49% of US nursing-home residents have a dementia diagnosis, from our own Spark count over 133,920 rows of federal CMS data for Q3 2026.

What it does

Nametags is a memory aid that quietly tells a person living with dementia who they're talking to, and gives them something warm to talk about. It's designed for smart glasses.

Recognizes people by voice and face. Every visitor gets one person ID that links their face, their voice, and everything the family has shared about them. Whether the camera sees them or the mic hears them, both lead to the same person, and the display tags them with their name and relationship.
Never guesses. If it isn't sure, it gives no name. A wrong name is worse than none.
Gives a warm context card. After each visit, Gemini turns the conversation into new facts about each visitor and a short card of things to ask about, e.g. "Last spoke about: graduating in Nov · Works in software · Loves cars | Ask about: the new Tesla, graduation plans." Cards never mention deaths, illness or bad news, and they're read straight from the database, so they work offline.
Visitors enroll themselves, with consent. A sign-up link walks them through consent, five face photos, a short voice recording and a few notes about themselves.

Who it's for: people with mild Alzheimer's, who still know their family but can't find the name. Moderate dementia is untested.

What it isn't: a treatment. Nametags is a non-drug support for the moments that matter, working alongside medical care, not instead of it.

How we built it

The stack and how it connects

Frontend (React, TypeScript, Vite): four pages served by the API.
- /visitor: consent, 5 face photos, a voice reading, notes.
- /setup: the wearer's name, notes and voice.
- /live: the glasses HUD with name tags, a card for each person, captions and Nova.
- /manage: edit people, cards, facts, photos and recordings.
Browser → API: the /live WebSocket streams camera frames and 16 kHz audio to the backend; sign up uploads go over HTTP.
Backend (Python, FastAPI, Pydantic): receives frames and audio and runs recognition.
- Faces: face_recognition and dlib turn each face into 128 numbers.
- Voices: Silero VAD finds speech, then TitaNet-small (run through sherpa-onnx) turns it into 192 numbers.
Database (PostgreSQL with pgvector): stores people and their face and voice embeddings.
- One SQL query finds the nearest two people.
- A name is given only when the match is under the cutoff and clearly ahead of the runner up.
Presence tracker: a face needs two sightings within 3 seconds, a voice counts at once, and each arrival is announced once ("Here's David, your grandson.").
Captions: Moonshine on the machine by default, or Speechmatics streaming if a key is set. Speaker labels are matched against voice samples in Postgres.
Context (Claude): notes and visit transcripts go in; strangers' lines are left out.
- One call writes neutral facts.
- A second call writes a short card for the HUD, with no bad news.
- Both are saved in Postgres.
Nova (Claude with tools): a voice assistant.
- Three tools (list_people, person_context, who_is_here) read only from Postgres.
- She answers "who's here?" and never claims to be a person.
Deployment (Docker Compose): a Postgres and pgvector container plus an API container that serves the built frontend
Privacy line: photos and audio are recognized on the machine. Only text goes to Claude, and audio goes to Speechmatics only if you add its key.
Research (PySpark):
- 5,804 Europe PMC abstracts and 133,920 rows of CMS nursing home data.
- 20 papers read in depth.
- The result was a ranked list of changes; four are built.

Challenges we faced

Voices in a real room: synthetic voices matched at 0.92 to 0.98 similarity, but real people on a laptop mic matched far lower. We loosened the live cutoff, judged speakers on their recent attempts only, and lengthened sign up recordings to 15 seconds.
Overlapping speech: Speechmatics gives one speaker per word, so two people talking at once merge.
Strangers' voices: they were first captioned as the wearer. Now they're "Someone" and kept out of summaries.
Camera and mic: browsers allow them only over HTTPS or on localhost. We also wrote our own resampling filter to keep hiss out of the voice band.
Testing limits: no users living with dementia, no real glasses, no CI. The care home interview recording came back silent.

Acknowledgements

HackPSU 2026 organizers.
Recognition: face_recognition, dlib, OpenCV, NumPy, Pillow, NVIDIA TitaNet-small, sherpa-onnx, Silero VAD, Moonshine, SpeechBrain, Supertonic.
AI and speech: Anthropic Claude, Speechmatics, Google Gemini.
App: PostgreSQL, pgvector, FastAPI, Pydantic, uvicorn, uv, pytest, Docker, React, Vite, TypeScript.
Research:
- Data and tools: Europe PMC, PubMed Central, Crossref, CMS Minimum Data Set, PySpark, matplotlib.
- Guidance and figures: Alzheimer's Association, WHO, Dementia UK.
- McCarron et al. 2019, the closest prior art.
The care home worker who shared how staff introduce people.

What we learned

Agree on the interface and ship a mock first, parallel work depends on it.
Tune recognition in the room you'll demo in, not on clean recordings.
Log match distances and timings, so tuning uses numbers instead of guesses.
Keep the factual record separate from what the wearer sees.

What's next

A pause mode for private rooms, and keeping everything on the machine in care homes.
Cue first, then the name: "Your grandson is here", then the name a moment later.
Settings for each person: announce on arrival, only when asked, or never.
Faster naming, using the timing logs.
Consent stored on the server; a login and HTTPS.
Name two people talking at once; one voice cutoff everywhere.
Test with people living with dementia, their carers, and real glasses.