vocabl.

Inspiration

Losing speech is far more common than most people realize. ALS, cerebral palsy, stroke, traumatic brain injury, and a range of developmental conditions can all leave someone with a fully intact mind and no reliable way to get words out. Millions of people worldwide need AAC (Augmentative and Alternative Communication) to take part in a conversation, and that group is badly served. So we set two constraints. The hardware had to be cheap enough that cost is a non-decision, a $3 analog joystick and a $5 microcontroller paired with any tablet that runs Chrome. And a full reply had to cost one selection instead of fifty, made with a movement a shaky or fatigued hand can actually produce: a flick of a joystick in one of four directions.

What it does

Listen. A caregiver talks into the mic, or types the line in “They said.” A programmable button can mute or unmute listening so the board is not always on.
Interpret. The last thing said is read with the user’s profile, recent turns, and saved facts in the background, so the next replies stay on this topic instead of a random memory.
Generate. A local proxy asks language models for short first-person replies that answer that line. If the APIs fail, simple heuristics still fill the wheel.
Select. Replies sit on a four-slice wheel. Move with the analog stick, arrow keys, or touch. A programmable select button (or a short joystick click, or Enter) speaks the highlight. Double-click (or R) asks for a fresh set. Hold the click to open custom phrases.
Output. The chosen line is spoken aloud. The mic pauses so the board does not transcribe itself.

Scanning mode walks the highlight for users who can only press select. A calibration wizard maps a noisy stick to clean directions.

The board shows the conversation as it happens (what they said, what vocabl. spoke) and keeps that history so the next replies can use recent turns. After a quiet stretch, it also stores a one-line summary and any new facts.

The mute/unmute and select buttons are programmable, so the same board can match how someone actually moves.

How we built it

Hardware. ESP32, analog joystick, and programmable buttons for mute/unmute and select. Bluetooth sends tilt as four directions, a short press as select, and a long press as hold. The board can blink when a selection goes through.

Web. React app in Chrome: Web Bluetooth, Web Speech recognition, a reply wheel, settings, and a typed fallback for “They said.” Aimed at a Windows tablet. Keyboard is a full backup (arrows, Enter or hold, L for the mic, R for new replies).

AI proxy. A small Node server keeps API keys off the tablet. It queries Grok for replies, handles text-to-speech, and writes a short memory line when a conversation ends.

Access. Custom phrases, volume, voice, scanning, and joystick calibration persist on the device. The same select path works for stick, programmable button, keyboard, and touch.

Challenges we ran into

A raw stick is not four clean directions. We had to debounce flicks, tell a click from a hold, and add calibration when rest is not the middle of the range.

Reply quality is easy to get wrong. Four paraphrases of “yes” are useless. Interpretation has to stay glued to the last sentence so memory does not hijack the topic.

Accomplishments that we're proud of

A loop you can show in one take: talk, wheel fills, select, tablet speaks, partner answers, repeat.

Programmable mute/unmute and select, plus the same actions on the stick and keyboard, including hold-for-custom and double-click-for-new-replies.

A wheel that is actually usable: replies stay inside the slice, and the caregiver can still see what was heard.

What we learned

The problem is not “they need more words on a screen.” For many nonverbal people with motor disabilities, spelling is too slow for a real conversation. Caregivers switch to yes/no. The person’s answers get smaller than what they could say if a turn only took a second. AAC has to keep up with talk in the room, not replace it with a keyboard.

We learned how to work with the hardware, get correct behavior out of a noisy analog stick and programmable mute/select buttons, and connect that to the software over Bluetooth.

What's next for vocabl.

First, test with real patients and their caregivers: nonverbal users with motor disabilities, in the rooms where they already talk, not only on a hackathon table. We need to see if listen, interpret, generate, select, and speak actually stay fast enough, and whether joystick plus programmable mute/select matches how they move. That feedback decides what we keep.