As we all know, people who communicates in sign language have hard time communicating with each other, so we created an video conferencing application for specially abled persons to have a better time communicating with others.
What it does
Here we have one of the person in the video as one who communicates in sign language, so we created an aid for him by converting the speech of the person to sign language.
How we built it
This project consists of a video conferencing app, a speech to text app as well as speech to sigh language. As we know that due to the digitisation of education, corporate, etc online conferencing is the best way to communicate. But, specially-abled people who cannot speak or listen...communication with and for them has become difficult. We have created a real-time web-app that uses socket.io for communication and also speech recognition API for speech to text. We have then taken that text and converted it into sign language. For this, we have used simple plain vanilla js with some ES6 functions. The speech to text transformation will be reflected on both sides i.e. listener's as well as speaker's side.
Challenges we ran into
A problem we faced was creating a peer js server for video conferencing.
Accomplishments that we're proud of
We are proud of that we built something that is useful for specially abled persons, and we could provide a ease for them communicating.
What we learned
We learned to build Deep learning model using CNN and LSTM, along with socket.io and peerjs for video conferencing
What's next for Breaking-Bad
In future, we are planning to implement it for multiple users, where users can acces therre converted sign language at a later time as well. Secondly we are planning to implement a Deep Learning CNN + LSTM Model for sign language recognition as well.