SilentVoice

• 

  Here you can see the gesture and the subtitles for "THis is"

Inspiration

I met a half-blind and deaf woman at school. I saw that she was standing in 1 place for a while. So I approached her and awkwardly tried to ask her if she needed some help. She took out her phone and started typing cause she understood that I didn't know the ASL(American Sign Language). She was asking for directions to get to the specific Hall. I showed the directions and she thanked me.

I think if we actually had a translator from ASL to english it would have been much easier for both of us interact and have conversations. And since she was half-blind, everytime I answered by typing something on her phone notes, she would have to look at the message through her magnifying glass. With the translator it would take seconds!

What it does

So I created a very first version of a SilentVoice - translator that translates sign language to subtitles in english in real time.

How we built it

That was honestly a long process since I had to choose Python instead of Java because Python had the libraries I needed for the video analysis. I used a webcam with OpenCV and MediaPipe fpr hand direction and landmark retrieval. Also I had to record myself as a model for the program to learn the gestures and their meanings.

Challenges we ran into

One thing I learned the hard way: when you're working with MediaPipe, make sure you're using an appropriate version of Python. I was stuck for two days debugging my code for import errors until I realized that I was running an unsupported version of Python 3.13, which was fixed by switching to an older supported version. Also, I had an old HP laptop that I got from my mother, it is quite an old version, so the camera was not as good as I needed for that project.

Moreover, since I used myself - a person whp barely knows any sign language gestures, the data might be not really accurate for people who use ASL. BUT! That is temporarily, because we can fix it easily by getting other people who really know ASL involved.

Accomplishments that we're proud of

I am proud that the program really worked! Cause at some point it wasn't working at all, all the bug logs were leading me to the wrong direction. I am also proud that it looks clean: the way hand gestures are highlited, and subtitles are added.

What we learned

I have learned how to work with Pythin and OpenCV, MediPipe and even a little bit of ASL!

What's next for SilentVoice

I really hope this project carries on. So I would want to make it dual : If right now we have translator from sign to english then we could make from english to sign language. Also, by collaborating, add this program to glasses, or make a special device that you can actually carry all the time! Later on we might add other common languages - spanish, etc but that is after we do what we described above.

Built With

• mediapipe
• opencv
• python