ASL Live Translator

Increases the independence for those who are hard of hearing by translating ASL to Live Captions during in-person conversations.

Comment

Inspiration

The inspiration behind the ASL Live Translator came from a desire to bridge communication gaps between the hearing and deaf communities. Many people struggle with language barriers, especially when it comes to sign language. This website aims to make communication more accessible and inclusive.

What it does

The ASL Live Translator uses a webcam to capture American Sign Language (ASL) signs, which are then converted into live captions. This allows users to communicate seamlessly with individuals who are deaf or hard of hearing by translating ASL into text.

How we built it

We built the website using Next.js, incorporating:

  • TensorFlow.js and the Handpose model for recognizing ASL signs.
  • Fingerpose for gesture estimation.
  • Webcam feed integration using the react-webcam library.
  • Text-to-Speech integration for real-time speech conversion. The project involves custom components like WebcamFeed, LiveCaptions, and SpeechSynthesis.

Challenges we ran into

  • Webcam integration was a bit tricky, especially with handling different camera states and permissions.
  • ASL recognition accuracy was challenging, as the model had to accurately capture and identify signs in varying lighting conditions and different sign variations. Ensuring real-time performance was a struggle with the amount of data processed during sign detection and translation.

Accomplishments that we're proud of

  • Successfully integrating real-time ASL translation to captions using machine learning.
  • Making the website responsive, clean, and accessible for different devices.
  • Ensuring the translation happens almost immediately with a user-friendly interface.

What we learned

  • Gained hands-on experience with TensorFlow.js and machine learning models for real-time applications.
  • Learned more about gesture recognition and its limitations in real-world usage.
  • Improved problem-solving skills related to real-time video processing and webcam integration.

What's next for ASL Live Translator

  • Expanding the range of ASL signs supported, including more complex gestures and phrases.
  • Integrating voice recognition to allow for a two-way communication flow between the user and the application.
  • Adding a feedback system to improve the accuracy of sign recognition based on user input and corrections.

Built With

Share this project:

Updates

Submission history