American Sign Language Detector

Inspiration

Not everyone can read or understand sign language, which can create a communication barrier between people who use sign language and those who do not. This inspired us to build a system that can recognize American Sign Language (ASL) and convert signs into something that can be understood by a wider audience.

Another major inspiration was TinyML. While building this project, we became interested in how machine-learning models can be optimized to run on small, resource-constrained devices. This project provides a practical path toward that research: eventually, the model could be optimized enough to run directly on smart glasses with a camera, recognize signs on-device, and convert them into spoken output.

What it does

The American Sign Language Detector recognizes ASL using computer vision and machine learning.

It has two main recognition modes:

• Static recognition: Recognizes hand shapes representing letters and numbers.
• Dynamic recognition: Analyzes sequences of hand movements to recognize movement-based signs and words.

The system captures frames from a camera, detects the hand, extracts its landmarks, normalizes them, and passes the resulting data to a trained classifier. The prediction is then displayed in real time.

The goal is to make sign-language recognition more accessible while creating a foundation for a future system that can work directly on wearable devices.

How I built it

We built the project using Python, OpenCV, MediaPipe, NumPy, and Scikit-learn.

The main pipeline is:

Camera → OpenCV → MediaPipe → Hand Landmarks → Normalization → ML Classifier → Sign Prediction

MediaPipe is used to detect the hand and extract its 21 hand landmarks. These landmarks are converted into numerical features and normalized so that the model focuses more on the hand's shape and structure rather than its position in the camera frame.

For static recognition, we trained a machine-learning classifier on landmark data representing ASL letters and numbers.

For dynamic recognition, we created sequential data from multiple frames and trained a separate model to recognize signs where movement over time is important.

We also created real-time inference and evaluation scripts to test the models and measure their performance using metrics such as accuracy, precision, recall, F1-score, and confusion matrices.

Challenges I ran into

One of the biggest challenges was generalization. A model can perform well on training data but struggle when the hand is viewed from a different angle, at a different distance, or under different lighting conditions.

Another challenge was extracting reliable landmarks from real-world images and videos. Some frames contained incomplete or undetected hands, which made sequence generation particularly difficult for dynamic signs.

Dynamic recognition was also more complicated than static recognition because a single frame is not enough to describe a movement-based sign. We had to work with sequences of frames while handling missing landmarks and inconsistent video data.

We also faced practical dataset challenges, including downloading and processing large video datasets and dealing with unavailable or problematic video sources.

Accomplishments that I'm proud of

We are proud of building a complete ASL recognition pipeline rather than only training an isolated machine-learning model.

The project can process camera input, detect hands using MediaPipe, extract and normalize landmarks, and perform real-time predictions using trained classifiers.

We are also proud of implementing both static and dynamic recognition, since they require different approaches to understanding sign language.

Another accomplishment was building an evaluation workflow that allows us to analyze model performance using multiple classification metrics instead of relying only on overall accuracy.

Most importantly, this project gave us a strong foundation for exploring TinyML and edge AI, where the same models could eventually be optimized to operate on small wearable devices.

What I learned

This project taught us that building a machine-learning application involves much more than training a model. Data collection, preprocessing, feature engineering, evaluation, inference, and handling real-world conditions are equally important.

We learned how to use MediaPipe for hand-landmark extraction, how to transform visual information into structured numerical data, and how machine-learning classifiers can be trained on those features.

We also learned the difference between static and sequential recognition and why temporal information is important for dynamic signs.

Most importantly, we learned how model performance can be affected by the quality and diversity of the data, and why testing on separate data is essential for understanding how well a model actually generalizes.

What's next for American Sign Language Detector

The next step is to improve the models' accuracy and robustness by using more diverse data and better handling of real-world variations.

We also want to explore TinyML and model optimization techniques so that the recognition system can eventually run efficiently on edge devices with limited computing resources.

Our long-term goal is to move beyond a computer-based camera application toward smart glasses with an integrated camera. The glasses could capture the user's surroundings, recognize sign language locally using a lightweight model, and convert recognized signs into spoken language.

This would turn the project from a desktop-based ASL detector into a more portable and practical assistive communication system.

Built With

• computer-vision
• deeplearning
• joblib
• machine-learning
• mediapipe
• numpy
• opencv
• python
• randomforest
• scikit-learn