-
-
webpage demo
Inspiration
This project aims to create an AI-based sign language to text translation device specifically designed for inclusive public service counters. The primary inspiration was to bridge the communication gap between individuals who are deaf or hard of hearing and public service staff, creating a more accessible environment.
What it does
To ensure the project remained cost-effective, the system operates entirely as a software application on a standard laptop, utilizing the built-in webcam to capture video. We built the pipeline locally on a Python 3.11 environment.
How we built it
Feature Extraction: We utilized Google's MediaPipe to track specific joints and landmarks of the hand in real-time. The system extracts exactly 63 hand landmark coordinates.
Mathematical Normalization: To ensure the model remains accurate regardless of how close the hand is to the camera, the extracted raw coordinates are normalized using the maximum Euclidean distance from the wrist (landmark $0$). This scaling process is driven by the spatial distance formula:
$$d = \max_{i=1}^{20} \sqrt{(x_i - x_0)^2 + (y_i - y_0)^2 + (z_i - z_0)^2}$$
Classification: This normalized 63-coordinate array is fed into a trained TensorFlow/Keras neural network for gesture classification.
Data Strategy: We used a hybrid approach, leveraging a pre-existing Kaggle ASL dataset for static alphabets and numbers, while using a custom Python/OpenCV script to record a dictionary of dynamic public service words (like "Help," "Form," and "Wait").
User Interface: Finally, we developed UI "stringing" logic to bridge individual signs together and output continuous, readable text.
Challenges we ran into
Setting up the environment presented significant hurdles. We initially used Python 3.13, which caused critical version conflicts because Google had not yet released the proper Windows C++ binaries for MediaPipe on that version. This resulted in an empty shell installation that lacked the 'solutions' attribute, alongside multiple NumPy C-extension import errors. We resolved this by completely rebuilding our virtual environment and downgrading to Python 3.11. Additionally, we faced overfitting issues during the Kaggle ASL static alphabet training, which we successfully mitigated by scaling the dataset to 1500 samples.
What we learned
Beyond gaining deep, hands-on experience with Computer Vision and Machine Learning frameworks, the most valuable lesson was the importance of project scoping. We learned that translating continuous, real-time sign language sentences is highly complex. By strategically narrowing our scope to static alphabets, numbers, and a 10-20 word custom dictionary, we successfully delivered a highly functional application. Ultimately, we proved that highly impactful accessibility tools can be engineered on a ₹0 budget without the need for expensive physical kiosks.
Log in or sign up for Devpost to join the conversation.