Silent Voice

Inspiration

Silent Voice was inspired by the communication barriers faced by Deaf and non-verbal individuals, especially in everyday situations where sign language is not widely understood. We wanted to create a solution that is not only effective but also accessible, without requiring expensive hardware or large-scale deep learning models, aligning with the goal of reducing inequalities in communication.

What it does

Silent Voice is a real-time sign language recognition system that translates hand gestures into natural language sentences. It captures hand movements through a webcam, recognizes gestures using temporal matching, and converts them into grammatically correct text, enabling smoother communication between sign language users and non-signers.

How we built it

We built Silent Voice using Python, OpenCV, MediaPipe, and NumPy. The system extracts 21 3D hand landmarks per frame using MediaPipe, forming a 126-dimensional feature vector for two hands. Gesture sequences are matched using Dynamic Time Warping (DTW), where the distance is computed as cost(qi,cj)=∥qi−cj∥2, and aligned through cumulative cost minimization. Each sign is stored as a template (.npy file), and the best match is selected in real time. The recognized words are then processed through a lightweight NLP component to generate grammatically correct sentences.

Challenges we ran into

One major challenge was the computational complexity of DTW O(N×M), which slows down performance as the number of gesture templates increases. We also faced difficulties with continuous sign recognition, as the system currently depends on clear start and end signals. Additionally, the model is sensitive to variations such as lighting, hand orientation, and camera angle, and it lacks non-manual features like facial expressions that are important in sign language.

Accomplishments that we're proud of

We are proud of building a fully functional real-time system without relying on heavy deep learning models or long training processes. The system runs on standard hardware, achieves reliable performance under normal conditions, and successfully demonstrates that lightweight approaches can still solve meaningful real-world problems.

What we learned

Through this project, we learned how to integrate computer vision, sequence modeling, and NLP into a complete pipeline. We gained a deeper understanding of Dynamic Time Warping and its practical applications, and realized that effective AI solutions do not always require complex models—efficiency and design choices are just as important.

What's next for Silent Voice

We plan to improve the system by transitioning to deep learning models such as LSTM or Transformers for better generalization, incorporating motion-based features like velocity and joint angles, and applying data augmentation to increase robustness. Future work also includes supporting continuous sign recognition, integrating text-to-speech and facial expression analysis, and deploying the system as a mobile or web application to maximize accessibility.