Sign Ease

Inspiration

In this current world where user comfort and accessibility are paramount to technology, there is still a large group of differently-abled individuals who are restrained from fully utilizing digital communication, specifically in the form of video calls, which is the current defacto form for communication from informal to formal use-cases. From communicating with your friends and family to partaking in online classes or meetings, no application presents sustainable options for mute individuals. Understanding sign language is a skill on its own, but when grouped with the digital environment, it only becomes more difficult. With this project, we aim to shoot an arrow towards the target of providing easy communication for mute individuals in the digital space.

What it does

Our project is a video-call web app, where the mute individual can sign comfortably. When the mute individual signs from one end of the call, it transcribes and provides subtitles in English in real time on the other end of the call.

How we built it

It was built with a combination of Node.js and React for the Frontend, while for the backend, we used a combination of OpenCV, mediapipe, and TensorFlow. Live video from the call is sent into the backend by the frontend, which uses pre-trained transformer models to recognize the sign, and send it back to the frontend.

Challenges we ran into

Creating smooth communication between the front and backend was a reoccurring problem. Choosing the best tradeoff between accuracy and time for the model was also a challenge we faced.

Accomplishments that we're proud of

We have created an application capable of ensuring accessibility for mute individuals, which makes us immensely proud. Being able to reduce the processing time to almost real-time is also an achievement in our books.

What we learned

Through this project, we have learned the depth of problems for mute individuals and the requirements of such individuals to make the digital space a more safe space for them, and for all of us. We also learned about teamwork and proper time and responsibility management.

What's next for Sign Ease

This program can be integrated into preexisting video conference apps like Zoom, Google Meet, or messenger and Whatsapp calls to make them more accessible. It can also be used as an extension to transcribe sign language in online videos. The main program itself can be launched as custom custom-tailored social media app for mute individuals.