Inspiration
Have you ever wanted to start learning sign-language? GiveMeASign.study is an interactive game you can play with friends and family. We built a "gaming console" using an Arduino UNO Q. The console has a small OLED screen that displays an alphabet letter. The user then has to sign gesture that letter in front of the camera on the web app. If they answer it correct, a green light turns on. Otherwise, a red light turns on when gesture detected is incorrect. The user's results are stored in the app, so that they can find out which letters they have trouble remembering the gesture for.
The app has two mode:
- a learning mode: where the user is shown the hand gesture, then they have to mimic it.
- a recall mode: where the user is shown a letter and has to remember what gesture to do.
What it does
GiveMeASign drills ten ASL alphabet letters to begin with: A, B, C, D, F, I, L, V, W, and Y.
Each player enters their name to track their own score. The laptop camera looks for a hand. Learn mode shows the letter and a hand diagram. Recall mode hides the diagram until the attempt is over. The player has to hold one letter until an amber bar fills, about a second. A match light green if player gestures correctly. A different light glows red if incorrect and then shows the diagram. Sudden hand movements don't accidentally trigger an answer submission, allowing players to take as much time needed to learn.
Scores stay with the player name. Switching players loads that person’s own record, letter by letter. If the Uno Q is unplugged, the same round still runs and the page draws the console itself.
How we built it
A computer vision model locally runs in the players web browser. MediaPipe’s Hand Landmarker finds 21 points on a hand image. Those points are shifted to the wrist, scaled to the palm, and rotated so the middle finger points up. A small scorer then rates each letter from finger lift, curl, thumb position, pinch, and spread, and keeps a letter only when its score is at least 0.62 and it leads the runner-up by 0.12. The model file is stored with the site, so the page works after the first download.
A FastAPI server on the laptop picks the next letter, stores players and attempts in SQLite, and posts the prompt and the result to the board. The Uno Q app listens on port 8765. Its sketch, through the Arduino router bridge, draws the letter on a 128×64 SSD1306 and drives a green LED on pin 12 and a red LED on pin 13. The laptop reaches that port with adb forward over the USB cable.
The public site is a Cloudflare Worker in front of a tunnel to 127.0.0.1:8000 on the demo laptop. Visitors use their own cameras.
Challenges we ran into
A pose that is almost a letter should not count. Early versions were happy to call a loose fist an A. Requiring both a high score and a gap over the next-best letter, then a deliberate hold, stopped random motion from lighting either LED.
The board could not depend on the hackathon network. Device discovery was crowded and often blocked, so the console stays on USB and the only link is the forwarded port. The game server binds to 127.0.0.1 for the same reason: localhost can resolve to IPv6 first and miss the process.
Putting that local demo on a public URL had the same constraint. The Worker forwards to the tunnel while the laptop is awake, and serves a fallback page when the tunnel is down, instead of Cloudflare’s error screen.
Accomplishments that we're proud of
One board serves every player. Names, modes, and per-letter stats are separate, and the on-screen console matches the OLED and the lamps when the board is not connected.
The public link and the table demo are the same game, with the hardware kept on the USB cable.
What we learned
Hand landmarks are enough for a small static alphabet if the scorer is picky. The hard part was refusing an answer, not producing one.
What's next for Team Registration
J and Z is a movement hand gesture, so those were beyond our means at the moment. Several other letters also look similar, so for this prototype we chose to implement a select few gestures.
Add the letters we left out, starting with the ones that differ by motion, J and Z, once a still pose is no longer the whole classifier. Then let a player drill only the letters they miss, using the per-letter stats the server already stores. .
Built With
- arduino
- c++
- cloudflare
- css3
- fastapi
- html5
- javascript
- mediapipe
- python
- sqlite
- vite
