SIGNify

Inspiration

People in the deaf community may have difficulties communicating with hearing people in day-to-day life because most people don’t understand sign language. There is a large language barrier, and simply writing down or typing what you want to say isn’t personal or smooth. Doing this lacks a lot of connection.

What it does

Our web app, SIGNify, provides a way for the deaf community to interact with the hearing more smoothly in real time. SIGNify allows users to translate each other’s form of communication in real time.

While the signer gets to sign to the camera, the receiver can read and/or listen to what they are signing as they watch, with the help of ElevenLabs. And to respond, the receiver can speak into the speech-to-text mode while the signer reads what they say.

The settings page enables users to change the languages they are translating to and from. Additionally, they can change the pitch and the quality of the voice using a simple slider.

How we built it

We are using a convolutional neural network combined with a bidirectional gated recurrent unit for the computer vision aspect. To model the exact motions of the users’ joints, we use Google MediaPipe’s Holistic Landmarker.

For sign-to-speech integration, we use the ElevenLabs WebSocket API, which transforms the signs into human-like voices. Our backend is in FastAPI, and our frontend is in HTML/JS.

Challenges we ran into

Relevant databases for this project were sparse and limited. There is no pre-built model for translation, so we had to build and train a machine learning model from scratch. Additionally, adding languages became challenging because the relevant databases for them were limited.

Accomplishments that we're proud of

SIGNify’s UI is sleek and minimalistic, offering accessibility and an intuitive layout. We tackled a difficult challenge that has seldom been approached.

What we learned

We learned how to incorporate different APIs into our app, including the Translate API and the ElevenLabs API. We also learned to code with Cursor for the first time.

What's next for SIGNify

We hope to expand the number of languages it supports and improve the model’s performance to better distinguish similar-looking signs.

Built With

• css
• elevenlabs
• fastapi
• google-gemini
• html
• javascript
• open-cv
• pytorch