Unlock ASL

An ASL learning and translation application, that can be used by anyone interested in learning ASL can be implemented in onboarding training for educators, hospitality, retail, and healthcare workers.

Comment 2

Inspiration

We made this application to help bridge the communication gap between hard of hearing individuals and hearing individuals to ensure fair and equal accessibility. In the US alone, 1 in 8 people are deaf or hard of hearing. ASL is the predominant language of deaf communities in the United States and most of Anglophone Canada. Especially at RIT, we have one of the largest deaf and hard of hearing student communities. We hope our application can be implemented on a basic level for ADA onboarding training in the education, hospitality, retail and healthcare industries. ASL is crucial for emergencies when hard of hearing people need to communicate, a basic knowledge of ASL by healthcare professionals can help transform a life threatening situation. ASL can also be beneficial for customer-facing employees in retail stores who deal with a wide range of customers on a daily basis. These employees need to anticipate these customers' needs and answer their questions which is something that is near impossible to do with a communication gap. The use of our application in onboarding training would not only create a more inclusive and accommodating environment for hard of hearing individuals. But would help companies become more inline with ADA standards of effective communication, preventing possible lawsuits action in the future.

What it does

We decided to make an interactive application that teaches users to learn ASL, with easy to navigate multiple choice questions. Multiple choice questions are an effective and simple way to measure learning where the learner is able to get prompt feedback. Users gain points for every correct answer and lose point for wrong answers. The amount of points gained or lost are determined by the user's confidence in their answer. Our application also has a translate function which is able to capture hand movements in real time and generate text accordingly. Using Twilio, Unlock ASL has the capability to text daily reminders to users to practice their ASL.

How we built it

Using Python, we coded a program that was able to capture more than 2,000 photos of one of our member's hands to use for reference. The photos were uploaded to Google Cloud's Vertex AI to build a dataset for the ASL alphabet. Using Google's AutoML (machine learning), a model was trained and deployed to an endpoint that could be accessed by our program. We were able to compare live camera feed using OpenCV, MediaPipe, and Tensorflow to identify various finger spelled ASL letters. Twilio was used to text users. We used PyCharm and VS Code to code and collaborated through git and Github.

Challenges we ran into

Since this was the first time any of our teammates worked with datasets and Google Cloud, there was a lot of time spent researching. We grossly underestimated the time the Google Cloud dataset took to upload, build, and train our AI even with our limited dataset. Due to the long compiling time of the Google Cloud services, we were not able to create a dataset beyond the ASL alphabet and we decided to focus on sign language for only one hand. We used the minimum recommended amount of about 100 photos per letter.

Accomplishments that we're proud of

  • Creating a website from scratch and formatting
  • One of our members learned a new coding language!
  • Creating and building our own dataset
  • Using computer vision to recognize various ASL words

What we learned

This was everyone on our teams first time using an AI system and machine learning so we started our project by learning how machine learning works and figuring our how we could implement it into our project. We where also unfamiliar with Google Cloud and spent a lot of time setting it up so we learned a lot about how it works and its capabilities. Our team also learned how to use ChatGBT and was able to advance our project much faster and efficiently.

What's next for Unlock ASL

In the future we will be adding to our database so we can teach and translate more than just the alphabet. We also want to have a "Onboarding Training Certificate" function that will give you a PDF certification of completion that you would then submit to employers as a part of ADA training. We also want to make our learning function more fun for our users. By adding a level system for harder questions we would also have different question forms. Like accessing the users camera and having them sign what the ASL translation for text. Or a video that the users use to complete a drag and drop translation question. We also would like to add a teach function, that would show the user a new word in ASL with the translation. After we taught them a new word we would then quiz them on it with a multiple choice drag and drop matching question.

Built With

Share this project:

Updates