Sign Language Interpreter

This project recognizes 3 sign language symbols and translates it into text for everyone to understand. These symbols are the ‘ok’ symbol, ‘thumbs up symbol’, and ‘c’ symbol shown below

Comment

See Live Demo here -> https://drive.google.com/file/d/1rZ4I1qkAv5pIwgwZPd15cMfwzIe2CEJg/view?usp=drive_link

Inspiration

This project is ML X Accessibility. In order to make the technological world more accessible for all kinds of people and help improve communication between them, this project recognizes 3 sign language symbols and translates it into text for everyone to understand. These symbols are the ‘ok’ symbol, ‘thumbs up symbol’, and ‘c’ symbol shown below.

What it does

It observes the hand symbols a user does and interprets the signals. Then it displays text on-screen explaining what the symbol.

How we built it

  1. Capture hand gesture images to train the model:

    • Used only 3 gestures but that can be extended to many.
    • Created many variations in the gestures to train more broadly.

  2. Prepare the images for MediaPipe compatibility:

    • Converted the images from BGR format (from OpenCV) to RGB format.
    • Imported hand detection landmarks from MediaPipe library.
    • Developed the logic to use only the landmarks for gesture detection.

  3. Train the model to detect each gesture using a classifier:

    • Used RandomForestClassifier.
    • Training accuracy reached was tested using a live video capture from OpenCV.
    • As the subject recreated the hand gestures, the model detected all the 3 gestures correctly.
    • Note that the model only detected 3 gestures.

Challenges we ran into

My code had difficulty running on Google Collaboratory and Kaggle. It took to long to load and use libraries.

Accomplishments that we're proud of

Capturing my own images to see how the model works was exciting!

What we learned

  1. Many libraries available that should be explored
  2. Several challenges in making the library work in different environments (Collab, Kaggle)
  3. Learned to use the very helpful library for recognizing hand gestures, Mediapipe.

What's next for The Sign Language Interpreter:

In the future, I hope to expand the scope to facial recognition and different patterns. I want to find niche use cases in research areas. We can also use it in the military. Satellite images can be observed to monitor its changes over the course of a few days.

Built With

Share this project:

Updates