SignQuest

The problem

Deaf and hard-of-hearing communities are underserved by learning resources, and most hearing people never learn even the ASL fingerspelling alphabet. Practising alone is hard because you get no feedback — you cannot tell whether your handshape is actually right.

What it does

SignQuest is a browser-based, game-style practice tool for the ASL fingerspelling alphabet. Using your webcam, it tracks your hand in real time, recognises the letter you are signing, and when you are close but wrong it tells you exactly what to fix ("Curl your ring finger more").

Four modes:

• Learn — match a reference hand with live finger-by-finger tutor hints
• Spell — 5-word rounds with speed bonuses and streak multipliers
• Speed — 60-second challenge with personal best tracking
• Weak spots — adaptive drills built from the letters you actually confuse

What makes it innovative

• Tutor feedback instead of right/wrong — per-letter templates and per-finger statistics become plain-language hints
• Ghost-hand overlay so learners can line their hand up with the target
• Personal confusion tracking — it learns that you show N when asked for M and drills that pair
• 100% client-side — no video leaves the device, no accounts, free to host

How we built it

• MediaPipe Hand Landmarker extracts 21 3D landmarks per frame (in the browser) and from dataset images (in Python)
• Landmarks are canonicalised (position, scale, aspect ratio, handedness) so the model ignores background, lighting and which hand you use
• A small MLP is trained on the landmark features with rotation/scale/noise augmentation, exported as JSON, and run with a hand-written forward pass in JavaScript
• Vanilla JS game engine with hold-to-confirm detection, 6-frame smoothing and localStorage progress

Challenges

Keeping Python training and browser inference numerically identical (verified to ~1e-16), handling image aspect ratio in landmark space, and making similar letters (M/N/S/T) usable by surfacing confusions instead of hiding them.

Limitations and what’s next

Fingerspelling only (not a translator), no motion letters (J, Z). Public datasets have few signers, so we plan to add on-device personalisation and collect data with Deaf educators and signers. Next we want to add dynamic signs and short words/phrases.

Built with

MediaPipe · JavaScript · Python · OpenCV · scikit-learn · NumPy · GitHub Pages

Built With

• css
• html
• javascript
• mediapipe
• numpy
• opencv
• python
• scikit-learn