Talk2Sign

Inspiration

466 million people have disabling hearing loss. 80% of deaf people are illiterate or semi-literate, most of them exclusively use sign language to communicate. Therefore, we need a app that help normal people talk with deaf people without spending years learning sign language.

What it does

It takes voice input and forms a transcript, and with proper prompt, it generates a series of sign language images.

How we built it

Backend: We used Python ML: whisper, Dall E from OpenAI API as pretrained models APP: We used streamlit to develop a fron-end template

Challenges we ran into

Dall E is not trained for sign language so the output image is not precise, we does not consider authorization problem that results in api leaking, hence the app can only run in local enviornment. As for sign language, dynamic animation or videos are preferred where Dall E is not capable yet. We are also lacking of dataset of sign languages.

Accomplishments that we're proud of

With the well-developed packages and models, api, we can easily develop a simple application without many coding experience.

What we learned

How to apply the pretrained models, utliize the tools to solve real world problems with current technology.

What's next for Talk2Sign

Have a strong authorziation system to prevent data leaking, personalize user experience with langchain, trained models or structure prompts properly to generate more precise outputs, restructure the back-end logic with existing techniques such as live time transcript to speed up the process and reduce the cost.