Brailed

Inspiration

Smartphones are the gateway to modern life — messaging, banking, navigation — yet blind and deaf users often can't use one privately. A blind user relying on a screen reader broadcasts every message out loud. A deaf user misses every voice message and call. Deafblind users are frequently locked out entirely unless a human assistant is present.

Refreshable braille displays solve part of this, but they cost $1,000+ because each tactile cell needs high-voltage piezo actuators (pin-based designs run on ~200V boost converters). We asked: what if we could deliver the independence — private typing, private reading, full phone control — using ~$15 of commodity hardware?

What it does

Brailed is a pocket-sized device that pairs with a smartphone over Bluetooth Low Energy and gives the user three capabilities:

1. Braille input — a 6-key chorded keypad (Perkins-style layout, one key per braille dot) lets the user type Grade 1 braille. Press a chord, release, and the character is decoded and sent to the phone.
2. Live captions out — the phone streams transcribed audio and incoming messages back to the device, which renders them as text on an OLED, so a deaf user can "read" what the phone is saying.
3. Agentic control — instead of memorizing menu trees, the user types a natural-language command in braille ("text mom I'm running late"), and an LLM interprets it and executes the corresponding phone action.

One device serves blind, deaf, and deafblind users.

How we built it

• Hardware: an ESP32 dev board, 6 capacitive touch sensors as the braille keys, a dedicated 7th push button for Send, and an I2C OLED display.
• Firmware: the ESP32 samples all six sensors and treats them as a 6-bit chord. We decode on release, not on press — a state machine accumulates dots while any key is held, then looks up the final bitmask in a braille table once all keys lift. This makes chording forgiving of imperfect finger timing.
• BLE protocol: a custom GATT service with an input characteristic (decoded characters/commands → phone) and an output characteristic (transcript text → OLED).
• Phone side: the companion app handles speech-to-text and passes command-mode input through an LLM pipeline (built with Jac/byLLM) that maps free-form text to concrete phone actions.

Challenges we ran into

Brownout resets. Our hardest bug wasn't code — it was physics. When BLE radio bursts coincided with OLED refreshes, the ESP32's supply voltage sagged and the brownout detector reset the chip in a loop. We diagnosed it by stripping the firmware down to a minimal BLE heartbeat sketch, then fixed it by cleaning up power delivery and reducing the CPU clock to lower peak draw — rather than the tempting-but-wrong fix of just disabling the brownout detector.

Send: chord or button? Our first design used "all six keys at once" as the Send gesture. In practice, landing six independent capacitive pads inside one debounce window is unreliable — six sensors means six chances for a false negative, and a silently failed Send is the worst possible failure mode for a user who can't visually confirm it. We ended up re-deriving a decision real Perkins braillers made decades ago: the most frequent actions deserve dedicated keys. Hence the 7th button.

Chord debouncing. Capacitive sensors are twitchy. Getting a clean, repeatable bitmask required tuning release windows so that slightly staggered finger lifts still register as one chord instead of two.

Accomplishments that we're proud of

• A full bidirectional pipeline that actually works — braille chord in, decoded character over BLE, live transcript back out to the OLED, end to end.
• Radical cost reduction — the core experience of a four-figure assistive device, rebuilt on roughly $15 of parts, making it realistic for users in low-income regions where assistive tech is least available.
• Debugging at the hardware level — tracing a "software" crash to voltage sag and solving it properly, without shortcuts like disabling the brownout detector.
• Designing for our users, not for us — choosing the dedicated Send button because a silent failure is unacceptable for someone who can't glance at a screen to confirm, even though the all-6 chord was the "cleaner" story.

What we learned

• Assistive tech design decisions have asymmetric failure costs: a mistyped letter is a typo; a silently dropped Send is a broken promise to someone who can't double-check the screen.
• Embedded debugging is often electrical, not logical — we learned to reach for a minimal reproduction sketch and a wall charger before rewriting code.
• Classic accessible hardware (the Perkins brailler) encodes decades of validated UX decisions. Studying it saved us from reinventing worse versions.
• How to design a clean bidirectional BLE protocol and keep an LLM in the loop for intent interpretation without adding latency to plain typing.

What's next for Brailed

• An optional refreshable braille cell on the output path, so deafblind users can read replies tactilely as well.
• Grade 2 (contracted) braille support for faster typing.
• Expanding the agentic command set — calendar, navigation, and app control beyond messaging.
• A 3D-printed case-mount enclosure so the device rides on the back of the phone as one unit.

Built With

• jac