Sign Language Translation

• 

  demo-frame_pt1

• 

  demo-frame_pt2

• 

  demo-frame_pt3

• 

  demo-frame_pt4

Inspiration

With presence of COVID-19, masks became the norm and fewer communication options were available to avoid physical contact. Thus we wanted to create a solution to help deaf people communicate better in this difficult time.

What it does

Takes a video input and classifies the person's actions (sign-language), predicting their associated words. The predicted words are then used to form a complete english sentence.

How we built it

We used a CNN-RNN architecture to build the sign language classifier and a word-level RNN to complete the word sentence.

Challenges we ran into

We lack data to build a robust sign language classifier.

Accomplishments that we're proud of

We are proud of developing a pipeline where the solution would definitely work, given that we had enough data to train the model effectively.

What we learned

We learned CNN, RNN and GRU architectures that were important in developing the models.

What's next for Sign Language Translation

Collecting more labelled data so that we can improve the robustness of the sign language classifier.

Built With

• beautiful-soup
• keras
• python
• selenium
• tensorflow
• yolov4