SignDeck

Inspiration

Learning sign language is hard, not because people don’t want to learn, but because there’s no immediate, accurate feedback when practicing alone. Most tools rely on static videos or quizzes, which don’t tell you whether your hands are actually forming the sign correctly.

We were inspired to bridge this gap by combining computer vision, machine learning, and spaced repetition system to create a system that helps users practice signs naturally, receive real-time response, and retain what they learn long-term. That idea became SignDeck.

What it does

SignDeck is an interactive sign-language learning platform that:

'Learn mode', where a reference image will show corresponding to a word and you will have to recreate it 3 times.

'Review mode', where you will be tested with a word being prompted without a reference image and have to recreate it.

Recognizes hand signs in real time using a machine-learning model.

Provides instant feedback on whether a sign was performed correctly.

Uses spaced repetition system (SRS) to schedule reviews and reinforce learning.

Tracks user progress, confidence, and mastery over time.

Users can practice signs using their camera, see which signs they struggle with, and review them intelligently just like a flashcard system, but powered by live gesture recognition.

How we built it

Computer Vision: MediaPipe is used to extract 3D hand landmarks from live video.

Machine Learning: A custom MLP neural network (PyTorch) classifies signs from normalized landmark features.

Backend: FastAPI handles inference, user data, and spaced repetition system logic.

Database: Supabase stores users, decks, cards, and review history.

Calibration & Filtering: Temperature scaling and confidence thresholds reduce false positives and improve robustness.

The system was designed to be modular, allowing future upgrades to temporal models (LSTM/GRU) for more dynamic signs.

Challenges we ran into

The biggest challenge we ran into was running into false positives without over fitting. In order to do this we calibrated confidence threshold and temperature scaling. This allowed us to mitigated false positives.