Sign Together

Inspiration

Fingerspelling practice usually means staring at a chart and hoping your hand is right — nobody checks, and nobody scores. For a hearing family learning alongside a Deaf child, that practice is also solitary, when it should be shared. A television is already the shared screen in the room, and a phone already has a camera. Sign Together puts those two devices to work: the TV shows one letter, and the phone checks the handshape.

What it does

Sign Together is a five-letter fingerspelling game for Amazon Fire TV. The letters are A, B, L, V, and Y.

The TV shows the target letter and keeps the score, from 0 to 5. The phone reads the hand locally and sends only three things: the guessed letter, a confidence value, and an event id. Camera frames never leave the phone. A point counts only when the letter matches the one on screen, and the same event is never scored twice.

It is not an ASL translator. It does not read sentences. J and Z are left out because they require motion.

How we built it

The television app is a Vega React Native app. The phone page runs in the browser and uses MediaPipe hand landmarks; a geometric classifier compares those landmarks with templates for the five letters. A prediction is sent only after four agreeing frames at a confidence of at least 0.72, followed by a short cooldown.

A local relay pairs the two devices with a six-character code. The phone page is served over HTTPS so iPhone Safari will allow the camera. The message format is the same one the AWS WebSocket template expects; that cloud stack is written but not deployed.

The Vega package builds. The simulator on this Mac did not boot, so the live round used a terminal stand-in that runs the same scoring code as the TV app.

Challenges we ran into

iPhone Safari blocked the camera on a normal http address. The relay had to serve an HTTPS page, and Safari still shows a warning for the local certificate.

The Fire TV simulator crashed during boot, so the scored round could not be played on a virtual television. No Fire TV was connected.

In the one live round, A and B were accepted on the first steady handshape. L and Y each missed once. V was much harder: the phone held a different letter steady 19 times before a matching V was accepted. That is one session, not a measured accuracy rate.

Accomplishments that we're proud of

A phone and the scoring app finished a real round at 5/5. The camera preview never left the phone. The TV, not the phone, decided the score. Repeated readings did not add extra points once a letter was already complete.

What we learned

The phone can recognize a handshape and still be the wrong place to keep score. The score has to belong to the screen that shows the target — otherwise a late reading of an old letter would count toward the new one.

A working camera permission is a secure-context problem, not just a model problem. And one successful round is evidence that the loop works, not a claim about how often the classifier is right.

What's next for Sign Together

Install the Vega package on a Fire TV — or a simulator that boots — and play the same round with the remote. Collect more camera tries before changing the V template. Deploy the AWS relay only after a fresh look at cost.

Built With

• fire
• html5
• javascript
• mediapipe
• native
• node.js
• react
• tv
• typescript
• vega
• websockets