Inspiration

Language is important and useful—but for some of us, it's not as easy as it sounds:

  • Out of every 1000 newborn babies, about 1 or 2 suffer from severe hearing loss.
  • Over 96% of congenitally deaf children worldwide are born to hearing parents, who are unfamiliar with sign language and have trouble communicating with their babies.
  • (WHO data on deafness and hearing loss, March 2019)

    We wanted to create an app that helps deaf children communicate better with the world, but sign language is more than just a tool for communication. Scientists found that learning sign language at an early age also helps with brain development and general intelligence in congenitally deaf infants.

    Through sign language education, we're providing a chance for them to develop into brilliant minds of the future, one Baby Sign at a time.

    What it does

    Baby Signs listens as the user speaks, and searches for the sign language translation in its sign language dictionary. The video "translation" is played after the user finishes talking.

    Baby Signs can be used mainly as an educational tool for children with hearing loss to learn sign language from their daily environment, or for parents to pick up sign language alongside their children. However, we also encourage people with normal hearing to learn sign language with Baby Signs. On the other hand, people who already know sign language can use the app as a way to understand spoken information (when other skills, such as lip reading aren't feasible).

    What's new about Baby Signs

    This is not the first speech-to-sign language project. However, Baby Signs is better:

  • Hand Talk is an app that converts Portuguese to Libras (Brazilian Sign Language) with cartoon animation.
  • Baby Signs translates English to American Sign Language, which is likely the most widely used sign language. For teaching purposes, we chose real-person recordings, which are more vivid and demonstrate digit movements more clearly.

  • Speech-to-Sign-Language-Translator translates recorded speech files to American Sign Language, but it is buggy and only has 159 words plus 26 English letters in its dictionary.
  • Baby Signs has a dictionary of over 8800 most commonly used words and phrases and works offline with real-time microphone input, so that parents can communicate with their children whenever and wherever they want.

    How we built it

    We acquired our text-to-sign dictionary from SigningSavvy.com and Handspeak.com, which in total contains more than 8800 words and phrases recorded in American Sign Language. We used PyAudio to stream microphone input and pass it on to Google Cloud Speech-to-Text API, which converts user speech to text. The text output is then fed into a python script that finds a good match in sign language: if the speech contains a phrase that exists in the dictionary, the phrase is preferred to its components; in this way we can avoid nonsense direct translations to a degree. Finally, the corresponding "translation" is played with a PyQt media player.

    Challenges we ran into

    We had problems deploying the Azure (microphone-)speech-to-text service (Speech SDK) on our laptops, even though the wav-to-text function works fine. After considering the alternative options, for example saving the user speech input as a .wav recording every time, we decided to use PyAudio to stream real-time microphone input and parse it with Google Cloud Speech-to-Text API.

    Accomplishments that we're proud of

    We came up with this idea, which we believe can actually benefit a large community and bridge the gap between individuals with and without normal hearing.

    What's next for Baby Signs

    What we have so far is a functioning prototype. Next, we want to enlarge our dictionary of sign language recordings, either by finding additional resources or by allowing uploads from sign language users. Second, our current method of generating sign language translations is still far from perfect. We might employ natural language processing techniques to create more accurate, context-dependent translations with linguistic nuances; an alternative would be to recruit fluent sign language users who could "proofread" the content. Finally, we might include illustrations and animations of cartoon characters (in addition to real person recordings) to make the app more appealing to our young users.

    Built With

    Share this project:

    Updates