According to the World Health Organization (WHO), over 5% of the world’s population suffers from disabling hearing loss. Five percent may seem like a small number, but that totals over 360 million people across the globe. The majority of these people live in low- and middle-income countries, where the access to healthcare and quality of health organizations is lower than that of the United States.

Sign language is the mostly used language by deaf and dumb/mute people that allows them to communicate without the means of acoustic sounds. But the main problem is that some people are not familiar with sign language and have difficulties in understanding the language and, Deaf and dumb/mute people have difficulties with communicating with normal people and expressing themselves to others whom they don’t understand, in a face-to-face situation. In that kind of scenario, they have to use some other level method like writing that and show, use primary sign language where common sense can understand. But all these methods are not proper methods to communicate and also its accuracy and performance isn’t good.

Also, on the other hand, deaf people face to a lot of problems in the current society. Since they can't hear anything or hear with loss, they cannot communicate with the normal people as normal which leads to reduce the social level and the quality of life.

What it does

The Smart Sign Glove, has a hardware glove and a hybrid mobile app. By the glove, it tracks the gestures of the fingers and movements of the hand and map it to the relevant letter or the word. It will then output that word/letter as a voice by the mobile app which allows the mute people to communicate with normal people as in normal way and also, the normal people to understand what the mute person says with sign language.

Apart from that the app supplies and interface which can be used by the deaf people to record/capture the speeches of normal people, and app will show it in the screen by text and also in sign language visually. This helps the normal public too, because they can learn sign language by this app.

How we built it

The glove includes five flex sensors (one per each finger) to track the movements of the fingers and an accelerometer to measure the movement of the hand. This glove connects to an arduino, where it get the input from these sensors, and send them to the mobile phone via bluetooth. Then, the app catches those data, translate them to the app-friendly format, and then by app it identifies the relative word or letter By using Machine learning (MLP), and it will output that letter and a voice using TTS.

For the function of listen service we used the Speech Recognition service, to grab the voice and translate it to the text. Also, we have a data map, which maps the text to sign image (Sign language's sign) and they will be shown at the view of the app.

For the arduino, we have coded it using C++ and for the mobile app, we used hybrid app framework Ionic. Javascript and angular. For the Machine Learning/Neural Network MLP we used ......

Challenges we ran into

The first challenge we ran into is the financial problems. For this prototype, we have spend around $200 for the sensors and boards and all, Since we are undergraduates - it was so hard for us to find the funds.

Working with Hardware is fun and also sometimes tricky. We had a lot of challenges with the glove like flex sensors and arduino to make them tweak to what we need :)

Also, in the machine learning part, we had hard time training the model.

Accomplishments that we're proud of

Though there are several projects on this scenario, there was no any project with all the features like we had. For example most projects are just take the sensor readings (Most of the time it's only flex sensors) and map it to the words. But we have added Machine Learning support to get better readings of the sensor data, interface to add user's custom signs, use of accelometer to track hand movements, translate the word by app as voice, help for the deaf people, learn sign languages by app and queue mode to use letters to express words (Highly fancy when it comes to express the names that are not traditional)

What we learned

A lot of hardware stuff, machine learning and team work + time management

What's next for Smart Sign Glove

Improve the hardware and the ML model

Built With

Share this project: