Inspiration
A blind user can hear a screen reader linearize a page, but they can't feel its shape — that a photo is a mountain range, that an article has three paragraphs left, that code is four levels deep, that a sine curve dips here and peaks there. Refreshable Braille displays are single-line and text-only, so the spatial half of computing is missing. We wanted to build the software layer for a different kind of device: a 32×32 grid of raised pins that renders layout, not just letters, with an agent that decides how to translate the digital world onto it for each type of input.
What it does
Send TactileOS text, an image, a math expression, or code, and it streams back a 32×32 elevation matrix plus an assertive ARIA description and a spoken caption.
- Text → Grade 2 Braille, with real word-signs, group-signs (th, ing, st…), number and capital signs, rendered in 2×3 cells
- Photo → a structure map (CLAHE → adaptive Canny + Sobel → area-pooled to 32×32) plus a spoken description of what's in the image
- UI screenshot → connected-component detection that labels tactile zones like header, nav, content, and button
- Math → auto-routed by type: y=f(x) as a curve, z=f(x,y) as a surface, or a bar chart
- Code → parsed with Python's ast for nesting depth and def/class caps, or bracket depth for other languages
There are two front-ends sharing one backend. A desktop workspace presents the 32×32 grid as an ARIA grid with a roving-tabindex cursor — arrows move, WASD pans, R scans a row — with each move playing a pitch-mapped Web Audio blip and pulsing a gamepad, plus a cloud-log inspector, session-history replay, and a Web Serial bridge to real hardware. A handheld "Braille Interface" offers an embossed Braille-cell readout and three keys — Capture, Ask, Listen — fully voice-first and screen-reader-narrated: capture a photo to feel and hear it, speak a topic and get an AI gist read aloud and paged in Braille with citations, or tap any cell to hear its character.
On the hardware side, there's a C++ HAL for a 74HC595 pin array, a framed serial protocol, and a byte-at-a-time parser that compiles for both Arduino and a host test harness.
How we built it
The backend is FastAPI on Python 3.11. A SpatialAgent picks a tool per payload — with a Gemini API key it uses Gemini for Google-Search-grounded article gists and vision captions; without one, a fully deterministic pipeline covers everything via Wikipedia summaries, structural heuristics, and a fixed type-to-tool map. Frames stream over SSE tagged by stage, and session memory lives in Firestore or a bounded in-memory ring with a replay endpoint. Every outbound fetch is SSRF-guarded.
The frontend is Next.js 14, React, and Tailwind, built around reusable pieces: a TactileMatrixDisplay, a SpatialAudioEngine with ADSR envelopes, a fluid embossed-SVG BrailleCellGrid, speech wrappers, and an /api rewrite proxy so the phone hits a single https origin — necessary since Web Speech and camera access both require a secure context.
Testing spans 90 backend tests (pytest), 110 frontend tests (Vitest), 7 end-to-end tests (Playwright), and a firmware smoke test, all running on GitHub Actions.
Challenges we ran into
- Model churn — the "current" Gemini flash model changed three times mid-build (fictional → retired → blocked for new keys). We added a model fallback chain, hard timeouts, and a demo-mode caption override.
- Web Speech needs a secure context — voice broke on a phone over a plain LAN-IP http connection; fixed by serving https and proxying the API through a single origin.
- Legible tactile contours — a naive bilinear downscale of a Canny edge map came out as noise; we fixed it with CLAHE, median-derived thresholds, a Sobel blend, area-pooling, and quantization.
- SSR/hydration — every browser capability check had to move into a post-mount effect.
- Config parsing and SSE under TestClient both needed real workarounds to test properly.
Accomplishments that we're proud of
The full loop actually closes: payload → agent → SSE → browser → Web Serial → firmware parser → HAL → physical pins. It runs with zero API keys, so nothing hard-fails on stage. The Grade 2 Braille, UI-region detection, and ast-based code analysis are all real, not mocked. Accessibility is architecture here — every visual element has an ARIA-live, audio, and haptic twin. And we shipped 207 automated tests with full CI on a hackathon timeline.
What we learned
Accessibility-first has to be a constraint on the architecture, not a coat of paint applied at the end. LLMs work best as the routing and description brain, not the renderer — the deterministic paths need to stay authoritative. And graceful degradation is a feature you build first, not one you bolt on later.
What's next for TactileOS
Driving real refreshable-pin hardware (the HAL and protocol are already built for it), an on-device vision model for fully offline captions, full Liblouis Grade 2 support across more languages, shared reading sessions, and larger matrices with haptic "peek" gestures.
Log in or sign up for Devpost to join the conversation.