Sign-Bridge

A quiz on a specific sign language lesson
This section allows a deaf person to record themselve doing sign gestures to be translated into text

Inspiration

Sign-Bridge was inspired by a personal experience within our team. One member has a Deaf brother who often feels excluded from family conversations due to communication barriers. This personal experience highlighted a broader issue: many Deaf individuals face social isolation because of difficulties in communicating with the hearing community. Motivated by this challenge, we developed Sign-Bridge to bridge the communication gap and foster inclusivity.

What it does

Our web application is designed to facilitate seamless communication between deaf and hearing individuals by enabling real-time sign language translation. Utilizing WebSockets technology, the platform ensures instantaneous, bidirectional data transmission, allowing users to engage in fluid conversations without noticeable delays.

Upon accessing the application, users can initiate a conversation where sign language inputs are captured via their device's camera. These visual gestures are then processed and translated into text or spoken language for the hearing participant. Conversely, spoken or typed responses from hearing users are converted into sign language, displayed through animated avatars or video outputs, ensuring both parties can comprehend and participate fully in the dialogue.

How we built it

eamless and responsive user experience. Recognizing the importance of building on existing innovations, we leveraged AWS GenASL to convert text into sign language animations.

The platform enables users to communicate effortlessly:

A user logs in and selects a friend who is deaf or hard of hearing to start a conversation. They can send a text message or voice note, which is then delivered to the recipient. Upon receiving the message, a request is sent to AWS GenASL, generating sign language gestures for the deaf user. On the other end, the deaf user can record themselves signing, and a pre-trained sign-to-text machine learning model translates their gestures into text. The translated text is then sent back to the original sender, completing the conversation loop.

Challenges we ran into

One of the biggest challenges we faced was the lack of pre-trained machine learning models capable of translating sign language into text and vice versa. This required us to build our own dataset, fine-tune custom models, and optimize real-time processing for accurate and efficient translations.

Accomplishments that we're proud of

We are incredibly proud that we successfully developed a functional prototype capable of real-time sign language translation. This is a significant step toward breaking down communication barriers, and we are excited about the potential to enhance and expand the platform in the future.

What we learned

Throughout this project, we gained valuable insights into machine learning, computer vision, and real-time data processing. We also deepened our understanding of accessibility challenges and the importance of inclusive technology in bridging communication gaps.

What's next for Sign-Bridge

Moving forward, we aim to integrate real-time video calling with built-in sign language translation, allowing users to communicate more naturally and fluidly. Additionally, we plan to refine our model, improve accuracy, and expand language support to make Sign-Bridge even more accessible and impactful.