According to World Health Organization, the population with disabling hearing losses are over 5% of the world's entire population and the estimated number will keep rise in future. The main communication methods for those with hearing disablity is through sign language. However, sign language is not well known to the majority of the public population. With the end product from this project, the user will be able to easily understand the sign language in different situations.
As a start, our team started looking into the hand and finger tracking for the input. For the base code, we looked at an open source code from a YouTuber called Nicholas Renotte [https://youtu.be/f7uBsb-0sGQ]. The base code was using webcam as an input source to recognize hand poses and output related emoji. We tried modifying for our project, but failed to compile it without an error. On top of the based code we were planning to use the sign language alphabet images to compare with the user inputs and identify what the input was.
For future works, we are looking into Google MediaPipe for hand tracking. It has more detailed output data format (finger joints locations with confidence points) and overall higher accuracy. The output data from Google MediaPipe then can be used to make a training labels for machine learning model for sign language recognition. We might be able to use Support Vector Machine (SVM) or a neural network model for this purpose.
Unfortunately, with limited experience and time, we weren't able to complete our project. As a result, there is no working video (we uploaded a blank video so that Devpost would allow the submission to go through). Yet we wish to build upon this experience and possibly continue on this project.
Log in or sign up for Devpost to join the conversation.