Inspiration
One of our teammates has autistic siblings who do not speak. Today they communicate by tapping picture buttons on a tablet. The tablet has to be in their hands, it is slow, and it is easy to leave behind. We wanted something that is always on the person and needs no screen at all.
What it does
Epsilon turns a wrist gesture into a message home. A non-speaking autistic child wears the Vexo band and makes a gesture they taught it themselves for water, food, parents, sleep, or help. Their parent gets a plain text message, such as "I want water, please." For an emergency word like help, the parent also gets a voice message that says when and how it was sent. A parent sees the whole conversation as chat bubbles on a Messages page, and chooses the words, the sentences, and which words count as emergencies. A family can also teach Epsilon a person's own sounds, or speak a word in their own way, and it suggests what it may mean for a person to confirm.
How we built it
We read the Vexo band's accelerometer and gyroscope in the browser. Each wearer teaches their own gestures by showing them a few times, and our own matching code compares each new movement with those examples. A recognised word goes to our server, where a fast Google Gemini model decides what to do with it: send a text, send an urgent voice message, or hold it back because the match is doubtful. It explains its choice in one sentence the parent can read. Gemini 4 Hydrogen reads the day's messages and writes a short summary with one practical suggestion. Gemini also listens to recordings of unclear speech and suggests which word was meant, and Deepgram writes down spoken words. Texts go out as iMessages through Photon, and ElevenLabs speaks the emergency messages. The app is plain Python and JavaScript and runs on Railway.
Challenges we ran into
The hardest problem was telling a deliberate gesture from ordinary movement. People move at different speeds and hold their wrists differently, so one fixed set of gestures does not work. The event band has only a motion sensor, a light, and a microphone, so every piece of feedback had to happen somewhere else. The event Wi-Fi blocked our laptops from reaching each other, so we moved the app to the cloud. Our first decision prompt quietly held back every message from the band, and on a silent recording the model once named a word with 95% confidence. We only found both by testing real cases, and fixed them with rules the model cannot break.
Accomplishments that we're proud of
It works from end to end. A gesture on the band becomes a real text on a real phone in about a second, with a voice message for emergencies. We are proudest of the part that stays quiet. A false message puts words in the mouth of someone who cannot easily correct it, so Epsilon holds back anything it is not sure about and says why. Two rules are fixed in code: an emergency word is never held back, and neither is a clear, new word.
What we learned
We learned how much calibration and testing matter with motion data, and that a model's decision needs hard limits around it. We also learned from research by and for autistic people and people who use communication aids. A tool like this should add to what a person already uses and never replace it. The wearer should control their own words. Stimming is not a problem to fix, so we never flag or count it.
What's next for Epsilon
Epsilon has not been tested with autistic users yet, and we have not measured how accurate the gestures are on a real wrist. Those come first. After that: a real phone call for emergencies, a model that learns each person's own sounds over time, letting the wearer see and delete their own message history, and moving the band connection from a laptop to the parent's phone.
Built With
- accelerometer
- deepgram
- elevenlabs
- gemini
- gesture-recognition
- google-deepmind
- gyroscope
- javascript
- motion-tracking
- photon
- python
- railway
- vexo-band
- web-bluetooth