Sign Language Interpreter Text to Speech

Inspiration

The inspiration for this project came from multiple instances of not understanding deaf people trying to communicate to use. This brought about the desire to bridge communication gaps between the hearing and non-hearing communities.

What it does

The Sign Language Interpreter Text to Speech application captures sign language gestures through a camera, processesing the images that the camera captures using a Roboflow AI model to recognize the letters or words being signed, using OpenAI to autocorrect the messed up letters and then converts these into spoken language using text-to-speech from the pyttsx3 library. This allows for real-time communication between sign language users and those who do not understand it.

How we built it

We built the application using a combination of computer vision technologies. We used multiple python libraries and two AI models, a roboflow ASL interpreter and OpenAI for autocorrect. The library OpenCV was used for capturing and processing video frames. After the images are processed, the recognized text is then passed through a text-to-speech engine, in this case the library pyttsx3, to produce audible speech.

Challenges we ran into

We ran into many problems, such as having an inadequate ASL interpreter model due to lack of time to train the model. We also ran into many bugs trying to combine the features and eliminating extra letters produced from the camera capturing too many frames.

Accomplishments that we're proud of

We are proud that this project has real life applications, rather than being used for leisure such as video games. This is also the first time we coded, so we feel like we achieved a lot.

What we learned

We learned more about how to use Roboflow and AI in general, and how AI models are trained and work. We also learned a lot of new syntax and libraries we can use, which was very interesting.

What's next for Sign Language Interpreter Text to Speech

Moving it to a smaller device like an Arduino so that it is easier to use in daily life for those who need it. Training the model better and getting a better autocorrect is also a next step, so that we can improve the accuracy rates of our interpreter.