Inspiration
My uncle lived with deafblindness. Conversations with him depended on a few family members who knew how to sign into his hands, and everyone else could only smile and wait. He was sharp, funny and curious, but most of the world couldn't reach him, and he couldn't reach it. We built Helen (named after Helen Keller) so someone like him could have a real conversation with anyone: a barista, a stranger at a bus stop, a person at a hackathon.
What it does
Helen gives a deafblind, non-speaking person a two-way conversation:
Hearing to touch: when someone speaks, Helen transcribes it, shortens it to the key words, and delivers it as vibration on a phone.
Touch to voice: the person replies by tapping on a Vexo wristband (simple tap counts or Morse), and Helen speaks the reply out loud in a natural ElevenLabs voice.
An AI turns a few tapped letters into a full sentence (COFE becomes "Could I get a coffee, please?"), so tapping stays fast.
How we built it
Vexo Open band: tap and wrist-turn gestures from its motion sensor, plus its microphone.
Python server: Deepgram live speech-to-text, OpenAI (Gemini 4 as backup) to expand taps into sentences and compress speech for touch, ElevenLabs for the voice.
Phone web app: vibrates the message, plays the voice, shows the conversation live. We also started a native SwiftUI iPhone app using Core Haptics.
Challenges we ran into
The band's built-in double-tap detection often split one double-tap into two taps, so we wrote our own tap pairing and filtered out "taps" caused by turning the wrist.
Live transcription in a loud hackathon room: we switched to word-gap detection instead of waiting for silence.
Stopping the system from hearing its own voice and replying to itself.
iPhone browsers limit vibration and mic access, so we designed around those limits.
Accomplishments that we're proud of
A full two-way conversation working end to end: speech arrives as vibration, and tapped replies are spoken out loud. Custom tap detection that is far more reliable than the band's built-in double-tap. It holds up in a loud room, and Helen never replies to its own voice. A few tapped letters become a full sentence, so conversation stays fast. Most of all, we built something my uncle could have used to talk to anyone.
What we learned
Accessibility is about timing and trust: a deafblind user needs confirmation that a message was sent, time to feel a reply, and a system that never talks over them. Small details like a pause before replying matter as much as the AI.
What's next for HellenKiller.ai
A phone-only version (no laptop), Braille-pattern haptics, a custom voice the user chooses as theirs, and testing with deafblind users and organizations.
