HandSync

Inspiration

There are 70 million deaf people in the world and only 2% of the population has access to a platform that helps them practice and learn sign language. Moreover, 90% of the parents don't know how to communicate with their dead kids. This creates a roadblock for kids, teenagers, and adults who are trying to learn and practice their sign language skills, as it may be hard for them to find peers, who know sign language, that can talk and practice with them.

What it does

Our website, HandSync helps provide a platform for learners to practice and hone their signing skills. By providing a room for two deaf people, with their videos on, they can start signing to each other. And, most importantly, the video provides a live translation for each sign that they make, allowing for the other person to not only understand what the person signing is saying but also learn from it and improve their vocabulary.

How we built it

We used WebRtc, a public API, to provide a peer-to-peer connection. Next, for the AI model that detects sign language, we used WLASL's vast dataset to train our video transformer model to recognize 2000 words. Afterward, we created a virtual webcam to overlay the text translation, allowing the other person in the chat room to see the word, live!

Challenges we ran into

Integrating WebRTC into our front end was a hassle as it can be tricky to set it up. However, we found out that Firebase allows for a much easier and seamless integration using a few lines of code after watching the tutorial. Next, creating the AI Model was painful as it needed a lot of GPU Power, time, configuration, and patience to obtain the best, most efficient model possible. Lastly, pushing our project to GitHub was also difficult as we encountered a lot of merging issues

Accomplishments that we're proud of

We are proud of the fact that anyone, from any part of the world can interact and talk to anybody in our platform as this allows for more knowledge and practical expereince to be learned

What we learned

We learnt how to work as a team and assign tasks accordingly according to our strengths.

What's next for Hack Sync

We plan to include More translations to make the website more accessible (French, Hindi, etc). We would also develop Beginner Courses and Videos & an AI Chat Bot to assist students for whatever they need.

Built With

• cv2
• next.js
• numpy
• python.
• pytorch
• torchvision
• transformers
• typescript
• vercel