Signul

Project Heading/Logo

Inspiration

Every passing day roughly 30 million Americans go through some sort of hearing loss. Growing up with classmates who were part of this statistic intrigued us into finding ways to help them express themselves in an easily accessible way. During our initial research we learned about various glove based approaches, that gather information for translation, as well as a project named "Sign Assist" created at UGAHacks 6 which uses Microsoft Kinect to track user movement in order to interpret the gestures. The issue with both of these became evident quicker than anticipated. Both methods are heavy on the wallet ranging anywhere from $50-$3500.

We created Signul as a cheaper, easy-to-access ASL detector that is usable by anyone with a webcam. Signul is an easily accessible alternative to many existing translators.

What it does

Signul is a real-time American sign language interpreter that uses any webcam to convert ASL letters and words into English.

How we built it

Signul uses a vision based approach. All the signs/letters are represented with bare hands, eliminating all external complications. We used OpenCV (Open-Source Computer vision) library to create our dataset. This was accomplished by gathering around 800 images of each ASL symbol in order to train the database, as well as 200 images each for testing purposes. Each frame is captured by presenting it to the webcam where a region of interest(ROI) is assigned using a blue square. The image is then used to extract the ROI which is then converted into a grayscale image. Finally, the gaussian blur filter is applied to our image which allows us to extract multiple features from our image. The gathered image is then compared to all the images in our testing database, which helps us classify it to the appropriate letter.

Challenges we ran into

There were many challenges faced by us during the project. The very first issue we faced was a dataset. We couldn’t find any existing dataset for that hence, we decided to make our own dataset. The second issue was to select a filter that we could apply to our images so that proper features of the images could be obtained and hence then we could provide that image as input for the CNN model. We tried various filters including binary threshold, canny edge detection, gaussian blur, etc. but finally, we settled with gaussian blur filter. More issues were faced relating to the accuracy of the model we trained in earlier phases which we eventually improved by increasing the input image size and also by improving the dataset.

Accomplishments that we're proud of

Heading into this project, none of the team members had little to no experience with python. We were able to create a product that can have real life implications when it comes to helping deaf individuals. With the database gradually expanding , Signul can be an accessible product used by people in need.

What we learned

We learned how to program with python, and build datasets used to HOI interfaces. We also learned how to implement OpenCV, CNN and Keras and many other softwares. We learned about training databases, which can be used to make the program more efficient in the future.

What's next for Signul

We plan on expanding Signul into adding speech, so that ASL is easily converted directly to speech. Signul will also be expanding its database with more variations in order to make it more efficient .