Signify

Signify is an educational app that uses Machine Learning and Computer Vision to detect ASL signs, allowing us to close the communication gap between people with hearing loss through gamified learning.

Comment

Inspiration

The WHO states that currently more than 1.5 billion people (nearly 20% of the global population) live with hearing loss. Over 5% of the world’s population – or 430 million people – require rehabilitation to address their disabling hearing loss (including 34 million children), and it is estimated that by 2050 over 700 million people – or 1 in every 10 people – will have disabling hearing loss.

This shows how grave of an issue hearing loss is, and one way we can play our part is through learning ASL so we can still communicate with the hard of hearing.

What it does

Signify is a gamified learning platform to support people with hearing disabilities, aimed to improve accessibility through providing a space where users can learn ASL and bridge the communication gap that they face. Users can study new letters of the ASL alphabet through the Learn system, which uses Computer Vision and Machine Learning to detect what letters the user is signing and checks if it is the same as a random alphabet flashcard shown on screen.

How we built it

First, I created collect_images.py to capture 70 images of me doing ASL symbols for each letter from my video feed using OpenCV, and labelled each folder with the letter. Then, I used Mediapipe and OpenCV to process the data and get the hand landmarks on each of the images and dumped the data into a pickle file.

Next, I reloaded the data with pickle and Numpy in a different file, and used scikit-learn to train the model with RandomForestClassifier and checked the accuracy of the model by checking the predicted data against the test data which was consistently 99-100%. After that, I made the realtime videofeed with hand landmark display, added the model's predictions as text, and created a backend server with Flask so that I could use this videofeed in my React app.

Then, I used React.js, Node.js, and Tailwind.css for the frontend of the web app so that it would have a more user-friendly UI, and I also used React.js to store variables for some of the game mechanics.

Challenges we ran into

Gathering a dataset for all letters of the alphabet in ASL to train the model took some time and the ML model still does not have perfect accuracy in the video feed, although it is able to map the predicted letter and test letter with 99-100% accuracy.

Some letters like 'k' are very easy to get the model to predict from the video feed, while others like 'n' are very difficult to get. This may be improved in the future with a more comprehensive dataset. Since I am a solo hacker this time around, I did not have time to fully complete all the features I wanted (like the time limit in the learning platform, translation, and creating a login and profile), but I am looking to add them after the hackathon!

I also experienced an issue with being unable to access data from the backend server in the frontend server due to the safety precautions taken by the developers, so I had to install the flask_cors flask extension.

Accomplishments that we're proud of

This was my first time creating a ML dataset and model, and training it myself. I am also very happy with how accurate the hand landmarks are after using Computer Vision for the first time as well!

What we learned

I learnt how to use a lot of new libraries like Tailwind, Flask, Scikit-learn, OpenCV, Mediapipe, and pickle and applied Machine Learning and Computer Vision in my project for the first time, which was a really exciting experience. I also learnt how to use React.js and Tailwind.css together for designing pretty frontends, which I really enjoyed.

What's next for Signify

Signify excited me since it has so much room for expansion, especially in terms of completing the translation from ASL to speech or text, etc. Right now, Signify only uses fingerspell (signing each letter of the alphabet) to communicate words, so one area for improvement is detecting actual phrases and words. Signify could also have a mobile version that would be more accessible for those on the go since it's more portable.

Share this project:

Updates