SignLingo

Slides

https://docs.google.com/presentation/d/10uj8WtSIYvAPtasd-Pbgb2xyQCztsYcvBby15iiVYxA/edit?usp=sharing

Inspiration

Learning the ASL fingerspelling alphabet from pictures is one thing; knowing whether you are forming the letters correctly is another. We wanted to bridge that gap with an interactive tool that lets learners practice using a webcam and receive immediate feedback.

That idea became SignLingo: a fingerspelling practice application that turns static reference material into an active experience. Instead of only looking at a hand diagram, users can try a sign, see what the app recognizes, and work toward spelling a word.

We kept our scope focused. ASL is a complete language with its own grammar, expressions, and cultural context. SignLingo is a fingerspelling practice prototype, not a complete ASL course or a replacement for learning from Deaf educators.

What it does

SignLingo uses a webcam to recognize hand shapes and provide feedback through two practice modes:

Learn Letters: Practice random letters or focus on a particular letter. Hold the requested sign while a progress bar fills, then move on after a successful match.
Spell Words: Fingerspell short words one letter at a time, with visual feedback showing completed letters and the current target.

Users can customize their experience through saved settings:

Tutorial mode: Show or hide example pictures of each letter.
Selfie view: Switch between mirrored and unmirrored camera input.
Show hand landmarks: Display or hide the hand-tracking dots without disabling recognition.
Low-power mode: Limit processing to approximately 12 frames per second.

The application processes webcam frames locally. It includes static-letter recognition and experimental movement-based recognition for J and Z.

How we built it

We built SignLingo in Python, using OpenCV for webcam capture and the interface, and MediaPipe to locate 21 landmarks on a hand.

For static-letter classification, we transform those landmarks into a compact feature vector:

$$ 21 \text{ landmarks} \times 2 \text{ coordinates} = 42 \text{ features}. $$

We make each point relative to the wrist and normalize it using the distance between the wrist and the middle finger’s base knuckle:

$$ \mathbf{f}_i = w_i \frac{\mathbf{p}_i-\mathbf{p}_0} {\lVert \mathbf{p}_9-\mathbf{p}_0\rVert_2}. $$

Here, $w_i$ gives fingertips extra weight because their positions help distinguish letters. This representation reduces sensitivity to hand position and size in the camera frame.

A $k$-nearest-neighbors classifier compares the resulting features with saved examples and lets up to five nearby examples vote on a prediction. Our reference data combines webcam recordings and landmarks extracted from an image dataset. We also generate mirrored references to support both hands.

J and Z require movement, so we added a separate recognizer that compares fingertip paths over time.

To keep the interface responsive, we moved camera capture and hand tracking onto background threads. We organized shared functionality into modules, added automated tests, and used PyInstaller to package the application into a Windows executable with its required resources.

Challenges we ran into

Distinguishing similar hand shapes. Some letters differ through subtle changes in finger placement. Camera angle, lighting, and tracking quality can make those differences harder to detect. Normalization, fingertip weighting, and additional examples helped, but broader accuracy testing remains necessary.

Recognizing movement. A single image cannot capture the defining motion of J or Z. We had to consider starting poses, movement duration, path shape, and mirrored directions. This feature remains experimental rather than a strict validator of correct signing.

Balancing responsiveness with reliable feedback. Background processing helped the display stay responsive, but it also meant the same recognition result could appear across multiple display frames. We added sequence tracking so repeated results would not count as new evidence toward completing a letter.

Making the application portable. Creating a usable executable required bundling the hand model, saved examples, letter PNGs, and native dependencies—not just the Python code. We tested the packaged application outside the repository to check that it could find its resources independently.

Integrating changes across branches. Interface improvements, settings, and recognition changes developed alongside one another. Resolving merge conflicts required preserving how these features worked together, rather than simply choosing one version of a file.

Accomplishments that we're proud of

We are proud of turning a hand-recognition experiment into a working desktop practice application with two modes, immediate feedback, and persistent settings.

Some milestones include:

Building a shared recognition pipeline used across the practice modes.
Supporting both static hand shapes and experimental motion-based letters.
Adding optional reference pictures through Tutorial mode.
Keeping recognition active when users hide the tracking overlay.
Improving responsiveness through background camera capture and tracking.
Passing 26 automated tests covering recognition-related behavior, settings, navigation, and threading.
Producing a Windows package and checking that both modes launch, return to the menu, and close successfully.

These accomplishments gave us a usable foundation while making the prototype’s remaining limitations clearer.

What we learned

We learned that a prediction is only one part of a useful application. Clear instructions, feedback timing, camera behavior, saved preferences, and reliable shutdown all affect the user experience.

Technically, the project taught us how feature engineering can make a relatively simple classifier useful. It also showed us when spatial information is insufficient and temporal information—how a hand moves—becomes essential.

Working with background threads taught us to separate display updates from new recognition results. Automated tests helped us check behavior without requiring a physical webcam for every test.

Most importantly, we learned to communicate limitations honestly. A classifier’s prediction does not prove that a sign is correct, and agreement among neighboring examples is not a calibrated accuracy score. Building responsibly means knowing what our system can demonstrate and what still needs validation.

What's next for SignLingo

Our next priority is to evaluate SignLingo with more users, cameras, and lighting conditions. We want to identify commonly confused letters, improve the diversity of our examples, and refine J/Z motion recognition.

We also want feedback from Deaf ASL users and educators to guide the accuracy and presentation of practice material.

Other next steps include:

Making recognition less sensitive to camera angle and hand orientation.
Expanding guided practice and progress tracking.
Independently building and testing the application on macOS.
Completing dataset and reference-image attribution and licensing review before wider distribution.

Our goal is to make SignLingo a more reliable and approachable fingerspelling practice companion while staying clear about its role within the much broader process of learning ASL.