Inspiration
Many extracurricular opportunities and learning programs in school do not cater to people with hearing disabilities, barring them from partaking in these activities. It's simply not fair that some people get a better shot at learning than others. This application aims to fix that, serving as a step forward in making this world fair for everyone.
What it does
This application can recognize letters in American Sign Language and output them, allowing people with hearing disabilities to finger-spell words. Once turned into an app, this program could be used at various events or programs to allow for students with hearing disabilities to communicate with other people.
How we built it
I used an ASL dataset from Roboflow, made by David Lee. Using this dataset, I was able to use transfer learning to retrain a YOLOv8 model. I then downloaded the weights from Google Colab and uploaded them into a PyCharm project. From there, I used OpenCV to access the webcam and run the model over the stream.
Challenges we ran into
This was my first time using transfer learning and YOLO, so finding my way around and debugging everything was very difficult. While training the model, I often had issues with memory and storage. To resolve the issue, I tinkered with batch size, image size, and number of epochs. Eventually, I was able to train my model and achieve a high accuracy. Another challenge I faced was implementing the finger spelling system. Due to the model often mistaking certain letters for others, I had to develop a system that would actually function well enough for words to be spelled. With the deadline approaching, I ended up implementing some simple logic and a button for erasing the wrong letters to work around this issue.
Accomplishments that we're proud of
I'm really proud of being able to create my own custom model for the first time. Seeing it work and with such a high accuracy was extremely satisfying for me. Also, getting it to work with the webcam was also very exciting.
What we learned
I learned so much about AI and the principles of computer vision. I gained valuable experience with OpenCV, YOLO, Roboflow, and Google Colab, which has set me up for so many future projects which I can't wait to build.
What's next for ASL Finger Spelling
Currently, the application is only capable of recognizing letters. The application cannot understand ASL phrases or words due to the fact that those are often represented by fluid motions and gestures. The model I created was trained on still images, and I didn't want to bite off more than I could chew by training a model on videos. Additionally, the framerate of the camera feed is very low, which can make it frustrating to finger spell at times. However, with more time and resources, this concept could easily be expanded into a fully fleshed out application.
Log in or sign up for Devpost to join the conversation.