Inspiration
Deaf and hard of hearing people live in a world where almost nobody around them signs. Most hearing people never learn any sign language. There's no easy place to learn, and no obvious first step. Classes take time and money, and apps just give you instructions without any actual feedback on if you're doing the sign correctly.
Without even a basic shared language, deaf people are always having to adapt to have basic human interaction.
A Silent Voice is something that changed my perspective. A film about a boy who bullies a deaf girl, then spends years learning sign language to make things right. What resonated with me was the relentless effort he put in, even at his lowest point he was still trying to make amends. Learning her language is how he shows he sees her. Rekindling their friendship started with a simple handshake, and that's what we aim to do. We wanted to make that first step of reaching out more accessible.
What it does
Handshake is a browser-based ASL fingerspelling app built around the person you're learning for. You enter your name, their name and your signing hand, and the app builds a path: your name first, then the alphabet, then spelling their name, then a few assisted greetings and a scripted first-hello scene.
During practice, the webcam compares your handshape with recorded references. Hand landmarks show exactly what's off (finger curl, direction or placement) and Gemini turns that measurement into a short coaching cue. If you keep missing, you get a different explanation. Practice words are built only from letters you've learned, with extra focus on the ones you struggle with.
A progress page shows your practice and match rates over the last 14 days. You can also opt in to weekly email recaps, or invite the person you're learning for to get lesson updates. Every recipient confirms by email first, and every message has an unsubscribe link.
How we built it
Front end (Vercel): React, TypeScript, Vite and Tailwind, with Three.js and React Three Fiber for the interactive 3D hand references, built from the average of recorded MediaPipe world landmarks.
Recognition (in the browser): MediaPipe gives 21 hand landmarks, which a One Euro filter smooths into 88 features. A TensorFlow.js MLP (trained offline with scikit-learn), or a nearest-neighbour fallback over recorded templates, classifies the handshape. A 10-frame majority vote and a ~0.8 s hold stop a single bad frame from counting. MediaPipe face and body landmarkers anchor the greeting signs. In dark rooms, OpenCV.js CLAHE brightens the frame.
Gemini (gemini-3.6-flash, @google/genai): MediaPipe measures your hands, and Gemini coaches you. Four Express endpoints use structured responseSchema output:
/api/coachrewords the landmark feedback into a short cue, and gives a different one on repeat misses./api/wordsbuilds practice words only from letters you know, weak letters first./api/explainexplains a mistake./api/praiseruns only after the app has confirmed a match.
Gemini only ever sees letters and error codes, never names or video, and never decides whether a sign passes. When Gemini is unavailable, canned fallbacks are labelled "cached". Live coach and words replies were confirmed against the public API. During our last check the free-tier quota was exhausted, so explain and praise ran on those fallbacks.
Tiger Data (TimescaleDB): an attempts hypertable, created with create_hypertable on ts. It stores anonymous attempt rows: letter or sign id, pass/fail, error codes and a random id. It feeds the #/stats page (14-day practice and match rates). Verified against the remote service (TimescaleDB 2.31) and through the public API.
Vultr: the Express API runs on a Vultr VM as a systemd service behind Caddy with automatic HTTPS on an sslip.io hostname. The browser calls it directly, with a CORS allow-list and per-IP rate limits. GitHub Actions tests each release and deploys it by commit SHA. Port 8787 is not publicly reachable.
n8n Cloud: four workflows handle subscription confirmation, a lesson-complete email to the person you're learning for, and a weekly recap. Gmail SMTP sends the emails, and PostgreSQL holds the subscription state. Consent is double opt-in: confirmation is a POST action so mail scanners can't grant it, and every email carries an unsubscribe link. We verified this on the live n8n runtime with controlled email sinks, including failure branches and a race test.
Problem-solution fit and community impact
Problem-solution fit. Each core feature answers one pain from the Inspiration. "Apps give instructions but no feedback" is answered by live webcam checking, where hand landmarks tell you which finger is off and Gemini rewords that into a short cue. "No obvious first step" is answered by a fixed path that starts with your own name and ends with a first hello to the person you're learning for. "Classes take time and money" is answered by a free, browser-only app with nothing to install. Our target user is a hearing person with a Deaf friend, relative or coworker. How well this fits deaf and hard of hearing learners themselves is still to be confirmed with them.
Community impact. The change we aim for is small and specific: a hearing person can fingerspell a name and say a basic hello, so the Deaf person in their life has to adapt a little less. Reaching out can also be shared, because the person you're learning for can opt in to lesson updates and see the effort being made. The limits are real. Fingerspelling is only part of ASL, and movement, facial expression and grammar matter too. Our references match our recordings and have not been checked by a fluent signer. We have no outcome data yet, and the app does not replace learning from Deaf teachers and the Deaf community.
Challenges we ran into
Recognizing handshapes across different people and different cameras was a hard task. Getting it to recognize the handshapes is easy, but when you incorporate movement in greetings, it can get hard to track, so we added MediaPipe's face and body landmarkers to find anchor points in facial and body position. Signs like NAME, NICE and MEET use both hands, and MediaPipe's left/right labels flip when the hands overlap. So we stopped trusting the labels and assign the dominant and other hand by their position on screen, swapping them back when the detector gets it wrong. The second hand is described relative to the first, in hand-widths, so the sign looks the same at any distance from the camera
Accomplishments that we're proud of
We brought a personal path, live hand tracking, interactive 3D references, picture guides and AI coaching into one prototype that starts with a reason to learn. We are proudest of being able to integrate CV into a web app through a hand-recognition pipeline in browser to give active rewarding feedback to the user.
What we learned
Building a recognizer and building a learning experience are different problems. A model output has to become a useful next step: which finger to move, what to look at, or when to try again. Matching a recorded handshape is also not the same as signing. Fingerspelling is only part of ASL. Movement, facial expression and grammar matter too.
What's next for Handshake
Move from letters to words and conversation. The path already goes from your name to greetings and a first hello. We want to grow that into more scenes and a larger vocabulary. That means more everyday signs and short reference clips of a real signer for each one, so learners see how a sign actually moves. Being able to take the time to 3D model entire sign expressions and conversations is also an idea we have for growth.
Built With
- gemini-api
- mediapipe
- n8n
- react
- tensorflow.js
- three.js
- tiger-data
- typescript
- vultr