-
-
Testing the ASL recognition!
-
Brian testing recognition in different lighting
-
First night starting the project!
Inspiration
Personal experience growing up with deaf parents and seeing many communication barriers inspired this project. Deaf, deafblind, and hard-of-hearing individuals deserve equitable accessibility to the world just like everybody else. For example, some Deaf, DeafBlind, and Hard-of-Hearing individuals are not familiar with technology, so we developed this so they could have a way to interact with the internet and text in a way that feels more intuitive to use. We aim to support and uplift them with this tool!
What it does
The program starts when the user hits the "Start" button. It uses scikit-learn and MediaPipe models to detect hand landmarks and classify the landmarks into the ASL alphabet into text in real time. Once the user puts their hands down, the text is deemed a sentence and output to a .txt file! The detection can be toggled on and off at the user's convenience.
How we built it
We began by developing the UI and researching how to classify hand landmarks. We used OpenCV and MediaPipe to build our UI and detect hand landmarks. We also researched user-friendly color palettes to ensure the application could be well-received and accessible for anybody!
We then integrated MediaPipe and a scikit-learn model to classify hand landmarks. Through this, we were able to build a dataset for each sign in the alphabet. Having integrated detection for the static signs, we focused on J and Z. As they are non-static signs, this presented the challenge of figuring out how to detect signs that act in motion. All other letters are static signs, but most ASL signs in general are non-static.
We were able to detect non-static signs through Mediapipe Preprocessing. Using 21 landmarks on the hand, coordinates are normalized relative to the image. We were able to detect 5 booleans, one per finger, to check how the fingers interact with each other and move in real time. We also checked for palm orientation and pinch distances to get a more accurate reading on what sign is being done!
We then used a 200-tree Random Forest Classifier to further improve the accuracy of predicted signs and check their confidence.
Challenges we ran into
Static vs non-static letters. Learning how to detect motion ranges for J and Z in the ASL alphabet.
Accomplishments that we're proud of
Getting the hand-gesture detection to work for both static and non-static signs! Having completed a project from start to finish at Bitcamp!
What we learned
GUI, ML, ASL recognition, Computer Vision.
What's next for SignToo
Integration for more ASL signs than just the alphabet, using the same methodology used for J and Z, as they are non-static signs!
Integration to MS Word, Google Docs, etc!
Log in or sign up for Devpost to join the conversation.