Visible Voice

Inclusion isn't asking them to change

Comment

Inspiration

Back in high school I had a friend named Jean. The entire time I knew her, she had cochlear implants - she could hear - but if they went dead, or there was a bug with the app, she lost her ability to communicate.

Because of this, I learned some basic sign language, nothing good enough to be fluent, but enough to exchange a joke across a room, enough to translate a message if everything stopped working, and enough to show I cared.

The entire time I knew her it was a running bit between us, not because my signing was perfect, but because the effort said something words never could - you matter enough for me to try.

That's what Visible Voice is built on. Not trying to fix deaf people - because they aren't broken, or trying to make inclusion a policy - because when a measure becomes a target, it ceases to be a good measure. The most powerful thing someone can do to show they care is to meet someone where they are.

Inclusion isn't asking them to change.

What It Does

Visible Voice is a fully offline VR sign language learning platform for the Meta Quest, built around three core modes.

Learn Mode lets you move through the entire ASL alphabet at your own pace, with clear, inspectable hand models showing exactly how each letter is formed. It's designed to build real muscle memory, not just visual recognition — with spaced repetition surfacing letters you struggle with more frequently, step-by-step breakdowns of the hardest signs (your R's, your X's, your E's), and a handedness toggle so left-handed signers aren't treated as an afterthought.

Quiz Mode is a speed-based challenge — think TypeRacer, but for your hands. Race to sign as many letters as you can in a set duration. Your streak grows the more consecutive days you practice, and the difficulty adapts around what you actually need to work on.

Translate Mode lets you sign freely while a live transcription bar at the top of your viewport shows what the model is reading in real time — as if you were signing in a real conversation. After each word, a Ghost Mode replay shows your actual hand position overlaid against the ideal reference, so you can see exactly where your form diverged and self-correct. No internet required. No latency. No waiting.

Everything runs locally on-device, fully offline — because the whole point of this app is to be there when it matters, and things that matter don't always happen somewhere with good WiFi.

How We Built It

Visible Voice runs on Unity targeting the Meta Quest 3, using passthrough so the real world stays visible while you sign.

The first approach used raw finger state comparisons — checking whether each finger was curled or extended and hard-matching that against expected configurations for each letter. It worked for simple signs, but fell apart fast. Several ASL letters depend on subtle wrist angle, thumb opposition, or partial bends that don't reduce cleanly to binary finger states. The approach had a ceiling, and that ceiling was too low.

The real breakthrough was switching to XRHandJointID. Instead of coarse finger states, the app now captures the full 3D position of every tracked joint on the hand — 26 points per frame — and feeds that data into a locally-running ML model trained to classify the ASL alphabet. The model learns the continuous, spatial shape of each sign rather than a rigid rule, which means it handles natural variation in form, accommodates left-handed signing, and generalizes the way a real human reading sign language would.

Beyond accuracy, the shift opens the door to words, phrases, and eventually fluency — because joint position data is expressive enough to encode motion and transition, not just static poses. The foundation is now built for what comes next.

Challenges We Ran Into

The hardware situation was, charitably, creative. Developing VR on a Linux laptop is not something Meta's SDK supports, which meant the primary development machine was immediately disqualified from running the build target. The Quest 2 that was initially available had significant passthrough limitations that made hand tracking unreliable in certain lighting conditions — a non-trivial problem for an app where seeing your hands is the entire point. Upgrading to the Quest 3 resolved the passthrough quality substantially, but added mid-sprint hardware context-switching to an already compressed timeline.

None of this was ideal. All of it got worked around.

Accomplishments We're Proud Of

Shipping a working, end-to-end product solo in a hackathon window is the baseline — but what stands out here is that it's a product that actually does something.

A lot of hackathon work lands in the category of could be useful if enough people adopted it. Poli.tech — which won a hackathon — fell into that category. It was built to change how people engage with politics, but its impact was contingent on scale and network effects that take years to build, if they ever arrive.

Visible Voice doesn't have that problem. Anyone who picks up a Quest and opens this app gets something real out of it immediately. They learn something. They practice something. And if they use it enough, they gain an ability they didn't have before — one that lets them connect with a Deaf person the way Jean deserved to be connected with. No adoption curve. No waiting for a community to form around it.

One person using this app is already a win.

What We Learned

Turns out if you sign into GitHub Desktop on someone else's Windows machine, it does not ask you to confirm your identity. It simply decides that all of your commits belong to whoever was logged in last. Every commit from the hackathon now reads as Nehemiah's contribution to ASL education — which is a great origin story for him, but not entirely accurate.

Thank you Nehemiah for the laptop. The commits are yours now. Congratulations.

Lesson learned: git config user.email before git push. Every time. On any machine that is not yours.

What's Next for Visible Voice

UI refinements, onboarding flow, and a branding pass to make the experience feel as considered as the technology underneath it.

Expanding the model to support words and common phrases beyond the alphabet — which the XRHandJointID foundation is already designed to support, since motion and transition data is captured alongside static pose.

Adaptive difficulty and speed tracking improvements in Quiz Mode, with metrics surfaced so learners can see their own progress over time.

Ghost Mode expansion beyond Translate Mode, so learners in all modes can compare their form against a reference after each sign.

Publishing to the Meta Horizon Store — free. The whole point is that barriers to entry should be as low as possible.

And as a final step before any broader release: Deaf-led content review. Having Deaf signers validate the letter models, catch regional variation the training data may have missed, and ensure the app represents ASL accurately rather than approximately. The inspiration for this project came from actually knowing and caring about a Deaf person. The least it can do is be correct.

Built With

Share this project:

Updates