ASL Interpreter Machine

Inspiration

We have been interested in Artificial Intelligence, and we would like to take this opportunity to start an AI project which we have never done before. Additionally, we've been thinking that knowing how to use sign language (e.g. ASL) can give us more opportunity to interact and understand more about the deaf and the mute communities, as well as the deaf-mute culture.

What it does

Our project - ASL Interpreter Machine - is based on computer vision, and we used OpenCV to build our program. The machine will take input from a video or the user's webcam, track the hand movement and face position of the ASL user in the video/webcam, and translate those movements by subtitles.

How we built it

Our program was built with Python, and as we said before, OpenCV - an open-source tool for computer vision such as hand detection, face detection, pose recognition, etc. - was used to help us program the project. After finishing the structure of the code, we collected the data (which are the photos of us doing sign language) and did the model training by Teachable Machine. Then, we downloaded the model and appended it into our code.

Challenges we ran into

Even though the program has an output, but the training is not really effective. Our machine misunderstood the user's action and displayed the different translation of the user's sign. In our opinion, because Teachable Machine only accepts images as samples, and the model was trained by stable images as well, it is hard for our machine to understand the moving input, which is the sign of the user.

Accomplishments that we're proud of

At least, we have achieve outputs, even though they were not perfect.

What we learned

While this is our first time doing a hackathon and encountering computer vision - a completely new field for us, we've learned at least the basic concept of OpenCV, and we also know some basic ASL phrases as well.

What's next for ASL Interpreter Machine

While the face and hand recognitions were not really efficient, we are thinking of doing a pose detection instead so that the translation accuracy can increase.