After watching a video about a deaf person being mocked and ridiculed at a drive thru, It made me open my eyes to the problems deaf people must face every day with communication. Therefore, I wanted to build something that could help them communicate more easily.
After many hours of brainstorming, I came up with the idea of a sign language translator that uses computer vision and machine learning to detect sign language gestures and translate them to english in real time
❓ What it does
It translates sign language gestures (only numbers for now) to english in real time using your computers webcam
💻 How we built it
I used open CV to build a dataset based on my hand gestures of over 5500 images on the numbers 1 to 10 in sign language
I then used tensor flow and keras to construct, train and test a convolutional neural network on the dataset of sign language number gestures
Then I used the live webcam and the trained model to detect and translate sign language numbers
💪 Challenges we ran into
I faced many challenges including small errors in my code that I spend hours looking for and changing as well as errors in the training of the model and losing lots of progress
I also faced challenges with the camera footage recording my webcam which wasn’t working, so I had to figure out how to use my phones camera as the webcam
I also faced challenges in the models training when the epochs would get skipped and the model only trained for 4/10 of the epochs
😀 Accomplishments that we're proud of
Coming into this project, as a beginner programmer, I knew almost nothing about Computer Vision and CNNs (Convolutional Neural Networks). I had to watch a video about it 5 times before I kind of understood a little about CNNs and had to work late into the night just to fix errors and get it done in time. However, I am glad I took part in this project because I learned a lot of useful skills I could use later on.
I was pleased of the final model which worked just as I expected (kind of)
🖐 What's next for Sign Language Detector
I aim on working more on, and improving the sign language detector to increase the accuracy and reduce the time taken to detect the gestures.
I also want to extend the functionality so that it can detect letters as well as numbers which I didn't have time to do in this hackathon
Log in or sign up for Devpost to join the conversation.