Inspiration
Communication is a fundamental human right, yet a significant barrier often exists between the deaf and hard-of-hearing community and the hearing world. We were inspired to build our AI Sign Language Detector after realizing how powerful technology can be in bridging this gap. We wanted to create something practical, accessible, and immediate a tool that empowers inclusive conversations and ensures everyone’s voice can be heard and understood effortlessly.
What it does
Our AI Sign Language Detector acts as a real-time communication bridge. By using a standard webcam, the app captures live video feed of a user making sign language gestures. The AI instantly processes the hand movements, recognizes the signs, and translates them into readable text and speech. This allows for seamless, back-and-forth communication between sign language users and non-signers without needing an intermediary interpreter.
How we built it
We developed our project using a combination of computer vision and machine learning frameworks:
Frontend: Built with a responsive interface to ensure users can easily interact with the detector in real time.
Backend & AI/ML: Utilized HTML and computer vision libraries to capture hand gestures, track landmarks, and classify sign language alphabet or words.
Processing: The model analyzes video frames frame-by-frame, calculating spatial coordinates and mapping hand gestures to text output.
Challenges we ran into: Real-Time Latency: Early on, our model lagged when processing video frames, making translations feel delayed. We had to improve our data pipeline and streamline landmark detection to achieve smooth, real time performance.
Lighting and Background Noise: Variations in room lighting and hand positioning initially threw off our classifier, requiring us to carefully refine our input handling and test under different conditions.
Accomplishments that we're proud of
Working Prototype: We successfully built and deployed a functioning end-to-end computer vision app from scratch within the hackathon timeframe.
Overcoming Latency: We managed to optimize our code so that the gesture recognition feels snappy and usable in real time.
Teamwork: As our first hackathon, we learned how to divide and conquer tasks, support each other, and push through bugs together!
What's next for AI Sign Language Detecter
Expand Vocabulary: Train the model on full sentences and complex phrases rather than individual letters or basic words.
Mobile Support: Package the app into a mobile version so users can carry a sign language translator right in their pockets.
Two-Way Translation: Add speech-to-sign capabilities to make the communication loop fully bidirectional.
Built With
- css
- html
- javascript
- mediapipe
Log in or sign up for Devpost to join the conversation.