Beacon

Inspiration

After a stroke, many people can't speak. Many can't control one hand. To press a call button in a hospital you have to find it, reach it, and press it. If you can't, you wait until someone happens to walk in.

Almost everyone can still make some movement: a finger curl, a wrist turn. But the gesture recognizers we tried all assume a typical hand, so a weak, shaking or half-closed one gets ignored.

So we flipped it. Instead of teaching the person a gesture, the system studies the person.

What it does

Teach it in about a minute. The patient holds whatever movement they can manage for 3 seconds per word: water, pain, help, bathroom, plus a rest pose. That builds a personal classifier in the browser.
Make the movement and hold it for about a second. A ring fills, then a request fires.
It speaks for them. The bedside unit says "I would like some water, please" in a natural ElevenLabs voice.
The caregiver is alerted. A Raspberry Pi with a Grove button, buzzer and LCD backlight turns red and beeps. A /caregiver inbox and a /beacon phone page work the same way.
One button to answer. The caregiver presses it, the unit turns green, and the patient hears "Help is coming." It then goes back to listening on its own.

What if there's no answer in 20 seconds? The caregiver gets an iMessage with a native poll ("On my way" or "Can't come"). They can tap it or reply in plain text, or text "status" for open requests, or ask a question and Gemini answers from the last 24 hours of requests (never medical advice). The first answer from any channel cancels the rest, and if someone presses the button after the text, the caregiver is told "already answered" so nobody comes twice.

It adapts as the patient changes. After a request, the patient screen can ask "Was that right?" (optional, never blocks help). Between sessions, a CREAO agent reads how the gestures performed and suggests changes in plain words, so the system keeps up when a patient's condition improves or worsens. The caregiver chooses whether to apply them.

How we built it

Recognizing movement. A camera tracks 21 points on the hand, like a tiny skeleton. We trained a small neural network on about 1.05 million public hand-pose landmarks (HaGRIDv2, 95% validation accuracy) and froze it. It turns the shape of a hand into a short list of numbers, an embedding. When a patient records examples of a movement, we store those embeddings. From then on, each new frame is compared to the stored ones with k-nearest-neighbours, and the closest match wins. Adding a new word, or a new patient just takes 5 examples (30s onboarding).

Connecting devices. The browser and the caregiver unit never talk to each other directly. They all talk to one server in the cloud (FastAPI and SQLite on Railway) and ask "anything new?" every second. We did this because venue Wi-Fi blocks device-to-device traffic.

Escalation. When a request fires, the bedside speaks it immediately. If nobody has pressed the button after 20 seconds, the caregiver gets an iMessage with a tap-to-answer poll. After 75 seconds, a phone call. Each step happens once, and the moment anyone answers, the rest is cancelled. The spoken phrases come from ElevenLabs which are generated ahead of time, and Photon delivers the iMessages.

Adapting to the person. Every session, the app notes which words fired cleanly, which were abandoned and which got mixed up. A CREAO agent reads those notes and suggests adjustments in plain words, such as "pain was confused with help four times; try a longer hold and re-record pain." The caregiver sees the suggestion and chooses to apply it or not. Every value the agent proposes is clamped to a safe range, so even a bad suggestion cannot break the system.

Watching it run. Sentry traces every request from the moment it fires until it is acknowledged, including how long the caregiver took to answer, with custom spans, Logs, Profiling, AI monitoring (Gemini token counts and CREAO spans) and releases.

Challenges we ran into

Noisy input. Accuracy stayed low until we stopped feeding the model jittery inputs and the always-zero wrist point
Photon is TypeScript-only and our server is Python, so we built a bridge. For a while, two running bridges both answered every message.
*Venue Wi-Fi blocked device-to-device traffic,* so the server became the hub and every device polls it.
The Grove LCD wants 5 V but the Pi's HAT supplies 3.3 V.
Gemini models were retired or rate-limited, so we built a 3-model fallback.

Accomplishments that we're proud of

A small hand movement in front of a laptop webcam becomes a spoken request and a buzzing red light on the caregiver's unit.
It learns a person in about a minute. In a few-shot test on 5 gesture classes held out of training, the frozen embedding reached 98.5% mean accuracy against 96.1% for raw landmarks (worst case 97.4% against 94.1%).
A robust voice-then-text escalation where the first answer from any layer stops the rest, and each layer can be switched off without breaking the others.
The agent's suggestions go to a human first, and every setting is kept within a safe range, so an agent failure cannot break the system.

What we learned

Learn the person, not the gesture. Asking "what can this patient already do?" made the product simpler and more humane.
A frozen embedding plus kNN transfers to unseen gestures better than raw landmarks, with no per-patient training. It is also faster and needs no training infrastructure.
Write the contract first. An hour of agreeing on the data formats saved us from merge conflicts and wasted time later.

What's next

Face input (head tilt, eyebrow raise) for people with no usable hand movement.
Two-hand gestures, which the embedding already supports.
Supervised testing with occupational therapists, and clinical review before any real-world use.