Silence

Inspiration

A conversation is two people taking turns. For someone who cannot speak, that turn-taking breaks: they type on a phone while the other person waits, or they are simply left out. We wanted the Vexo band to give that turn back, so a person can answer with their hands and be heard out loud, and still follow what is said to them.

What it does

Silence is a two-way conversation on one screen.

You draw, it speaks. Make a fist in front of the laptop camera and draw in the air. Open your hand when you are done. The drawing becomes one sentence, spoken out loud in a natural voice.
It answers what was just said. The sentence is written as a reply to the conversation so far. A cup drawn after "What can I get you?" becomes "I would like a coffee, please."
Their words come back as text. The band's microphone hears the other person, and what they say appears in large type on the screen.
The band is a second input. A slow or a fast shake of the wrist says a ready phrase such as yes or no, with nothing to teach.
It stays quiet when unsure. If a drawing is unclear, nothing is said. A wrong sentence spoken for someone is worse than silence.

How we built it

Band: the Vexo Open band over Web Bluetooth, straight from the browser. We read motion at 50 samples a second and the microphone as 16 kHz Opus packets.
Hand tracking: Google's MediaPipe gesture recogniser runs inside the page. We measure the finger joints to tell a closed fist (draw) from an open hand (stop). The camera picture is never shown and never leaves the laptop.
Shapes: five simple shapes are matched against fixed templates with a point-cloud recogniser, so they work at once.
Meaning: any other drawing goes to OpenAI as a plain line drawing, together with the last few lines of the conversation, and comes back as one first-person sentence.
Voice: ElevenLabs speaks the sentence.
Captions: the band's audio streams to Deepgram and the text appears as the other person talks.
Server: a small Python server on the laptop serves the page and keeps every API key off the page.

Challenges we ran into

Real wrists are not test data. Our first recogniser was tuned on the kit's pretend band. On a real arm, every movement pushed the sensor by about 1 g and half of the "gestures" it cut out were small twitches.
A language model could not read motion numbers. We tried describing each wrist movement in numbers and asking a model to name it. It gave different answers for the same recording, so we dropped that.
The band cannot see fingers. A wrist sensor feels the arm, not the hand shape. That is why drawing moved to the camera, with the band helping when a fast hand blurs.
Sentences that said too much. The first version turned a drawing of a house into a story. We now ground the sentence in what was drawn and what was just said.
One connection at a time. The band accepts a single Bluetooth connection, which cost us time whenever an old browser tab was still holding it.

Accomplishments that we're proud of

A complete loop that works in a browser with no app to install: hand to voice in one direction, voice to text in the other, on one screen that both people can read.

What we learned

Start from real sensor data, not simulated data. Pick the sensor that can actually see the thing you care about. And for a tool that speaks for someone, being silent when unsure matters as much as being right.

What's next for Silence

Words and letters written in the air, not only drawings.
Running the sentence step on the device, so nothing leaves the laptop.
Testing with people who use sign language or cannot speak, and learning what they need. Silence is not a sign language translator today.