ASL for the Community

Project Overview

Inspiration

We were inspired to help provide a way to make communication between the deaf and those of hearing an easy learning experience for all. We believe the image recognition combined with instant verbal feedback could allow ASL to be easily available to anybody in the Lehigh Valley.

What it does

Our project uses an image recognition model we trained in Amazon Sage Maker to identify ASL signs displayed Infront of the user's camera. Once identified, the sign is then translated to its corresponding letter, displayed, and spoken using Amazon Polly.

How we built it

Most of our creation of this program took place on Amazon Sage Maker, working as a team to best train our recognition model. Once we had a base model ready, we had begun to develop a camera feed which used said model to output the displayed letter. During this process, we worked on integrating Amazon Polly, along with developing the output of letters using Turtle Graphics. Once each of these portions of our project were created, we worked on piecing it together and improving the image recognition

Challenges we ran into

One of the largest issues we ran into was training our image recognition model in the allocated amount of time. This required us to cut down the supported letters of the program to A, B, C, as we were able to still provide a large amount of data for each letter, while being trained in a reasonable amount of time.

Accomplishments that we're proud of

Our entire group is proud of the overall successful completion of the project. While it was not to the scale we first imagined it as, we are very satisfied with the final product we ended up with as this was one of the first times any of us have taken part in a project similar to this.

What we learned

Our group has gained lots of new experience across the board, but one of the largest things we learned was in that of training a data model. The immense scale of data that is required to train a successful model was much more than we had expected, along with the training time for said model. We had also learned some information about front-end programming, which none of us had any experience with.

What's next for ASL for the Community

This project is very much still in it's infancy, and has many improvements across the board which could be done to it. The first steps would be allowing compatibility with the entire ASL alphabet and using more data to make it's detection more reliable. We would also like to possibly give this program a more presentable front-end interface once we gain the experience.