Visuall Motion

Standard phone calls remain one of the biggest accessibility barriers for Deaf and Hard-of-Hearing individuals, often requiring cumbersome third-party relay setups. Visuall Motion changes that.

Comment

Inspiration

Phone calls are one of the last everyday things that are still hard to make accessible. Relay services exist, but they are slow, they put a stranger in the middle of a private conversation, and they are a poor fit for the calls that matter most — a doctor's office, a landlord, a pharmacy. The barrier is not that the technology is impossible. It is that nobody has put signing where the call already is.

What it does

Visuall Motion puts a signing avatar into a live phone call, in both directions.

  • Inbound: someone calls a Vonage number. Vonage's ASR returns each utterance as text, the backend pushes it to the browser over a WebSocket, and a 3D avatar signs it.
  • Outbound: you type a reply and Vonage speaks it into the call as text-to-speech. The caller never needs an app, an account, or any awareness that the other end is not speaking.

The signing itself is two layers. Seventeen words are signed as whole words — HELLO, HI, HOW ARE YOU, YES, NO, THANK YOU, HELP, NEED, DOCTOR, TODAY, TOMORROW, COLD, ALWAYS, ALRIGHT/OKAY, ALL OF A SUDDEN/OUT OF THE BLUE. Anything outside that vocabulary is fingerspelled letter by letter through a complete 26-handshape manual alphabet, including the two traced letters, J and Z, that move through space rather than holding a shape.

What it is not, said plainly. This transliterates speech into signs in English word order. It is not full ASL. ASL has its own grammar, spatial syntax and grammatically-meaningful facial expressions, and my real-time ASL translation is an unsolved research problem — not a weekend of extra work. What runs here is closer to Signed Exact English plus a fingerspelling fallback. That sentence is in the UI itself, under the avatar. I'm aiming to not overclaim what this accessibility does for the people it claims to serve, so the app says what it does and what it does not and is not buried in a README.

How I built it

The call. Node and Express behind the Vonage Voice API. GET /voice/answer returns an NCCO that greets the caller and opens an input action; Vonage runs ASR on each utterance and POSTs the final transcript back. A ws server streams those transcripts to every connected browser. Turn-based ASR was a deliberate choice over streaming STT — a final transcript is reliable on camera, and it sidesteps partial-transcript reconciliation entirely.

The avatar. three.js and Vite. A Mixamo-rigged humanoid, driven bone by bone. The 26 handshapes are stored as tunable per-joint curl data rather than baked clips, so a shape can be adjusted numerically instead of re-exported. Exactly one file in the project touches bones; everything else describes poses.

The signs. Every whole-word sign was hand-authored in Blender, pose by pose, then exported as absolute joint angles and replayed in the browser. They were referenced against Lifeprint (ASL University, Dr. Bill Vicars, a Deaf ASL educator) and video demonstrations, and three people who know sign language — one of them hard of hearing — advised on how the signs are made and reviewed the avatar's playback.

The seam that made it work. The renderer never learns where text came from. A TokenSource interface sits between them, so the same signer runs off a live Vonage call, off a typed rehearsal box, or off a replayed transcript. Swapping Vonage's turn-based ASR for streaming STT later means writing one new TokenSource, not a rewrite.

Challenges I ran into

Two rotation orders that look identical. three.js composes Euler angles as Rx·Ry·Rz. Blender's "XYZ" composes them in the opposite order. So the same three numbers describe two different poses, silently, with no error anywhere — a sign that was correct in Blender put the wrist 15cm from where the browser put it. The fix is one line (hold the arm bones in Blender's ZYX, which composes the way three.js's XYZ does), and finding it took far longer than fixing it.

A wrist that was quietly being clamped. The letter J traces a hook through space. Its wrist rotation reaches 3.02 radians — but letters route through a safety clamp that caps wrist twist at 1.30, so no handshape can break the avatar's wrist. Everything past that limit was being flattened to nothing, and the last third of the J simply had no motion. Nothing errored. Fixing it meant giving traced letters the same storage format whole words already used: absolute joint angles per key, applied on the unclamped path.

A frozen left wrist in every two-handed sign. One object was constructed without a back-reference, and the left wrist stayed locked at its idle rotation. I checked the arm chain and the wrist's world position, found both correct, and said so — twice. Both checks were blind to it, because a wrist's position is set by the three bones above it, not by the wrist. A full 67-bone diff against the .blend found it in one pass.

Trusting my eyes. Screenshots of a WebGL canvas lied repeatedly — a background tab suspends the render loop, so the canvas keeps showing a stale frame that looks alive. Poses ended up verified by reading bone rotations back and comparing them numerically against the source .blend, to three decimals. That caught real bugs. It was also worthless for the other half of the problem: three things flagged as defects by measurement turned out to be fine when a human watched the playback, and none of them were faults.

Accomplishments that I'm proud of

It works on real calls, both directions. Not a mock, not a recording — dial the number, talk, and the avatar signs what you said.

  • 26 letters and 17 whole-word signs, hand-authored. No motion capture, no purchased clip library. Every one was posed by hand and checked against a reference.
  • The honesty. The UI states that this is transliteration and not ASL grammar. The connection pill says backend offline rather than faking it, because a demo that looks live when it isn't is worse than one that admits it. Neither of those makes the demo look better — both make it true.
  • It brought in people who actually sign. The signs are not guesses off a diagram.

What I learned

That measurement and judgment are different tools and neither substitutes for the other. Numbers found the bugs that were provably wrong — the flipped rotation order, the clamped wrist, the frozen bone — and none of those were visible by eye. But every time measurement was used to decide whether something looked wrong, it was wrong: clipping that was not clipping, a dead hold that read fine, an alignment that was already right. For a signing avatar the question is always "is this legible to a person", and the only instrument for that is a person. So i had to trust my eyes and the eyes of my resources and resourceful friends.

And that scoping honestly was the best design decision I could make, not a disclaimer. Deciding early that this transliterates rather than translates ASL is what made the rest of it buildable, and saying so in the UI is what makes it usable rather than misleading.

What's next for Visuall Motion

  • ASL grammar. Spatial syntax and non-manual markers — the facial expressions that carry grammatical meaning — are the difference between transliteration and translation. This is the real roadmap, and the hard part.
  • Formal review. Review with an ASL program coordinator is wanted and was deferred past this build, not cancelled. The current signs are a well-referenced, informally reviewed first pass.
  • Streaming STT, for signing that keeps up mid-sentence instead of per utterance. The TokenSource boundary already exists for it.
  • More vocabulary. Every word with a whole-word sign is one fewer word spelled out letter by letter, and fingerspelling is the slow path.
  • Authentication and a permanent deployment, the moment anyone other than myself the developer on this project needs to reach the service. The backend is deliberately unauthenticated and unhosted for this submission — the demo is a recording, so nothing has to stay up for anyone, and a permanent public URL would be less safe than one that dies when the tunnel stops. The plan is written down and waiting.

Built With

Share this project:

Updates

Submission history