tASL

Inspiration

The goal was to make everyday conversations at school, work, healthcare, and public spaces more accessible, inclusive, and effortless for everyone. Since no such application existed that could do this because we thought it was a great idea as it is something America needs.

What it does

Using video input, it converts ASL into clear English text, and it also translates English back into ASL letters for easy communication. It also can act like a personal tutor using the AI learning.

How we built it

We built it using the help of Google Gemini. It helped us with every step and without using it we would have never been able to finish this big of a project in such a little time.

Challenges we ran into

We ran into challenges where we did not agree with the AI's choices in parts of the application. We also ran into a problem with it recognizing ASL as we needed to train it to be able to pick up on it. We wanted a easy way for the videos to be able to be added but did not want to compromise how the AI analyzes the sign language. For this reason we had to set up a max of how long the video can be.

Accomplishments that we're proud of

We are proud that we got this to work. We did not expect to be able to accomplish this. We were happy if we were only able to implement it translating English into ASL letters. However, it being able to scan videos so the user can understand what the person is saying is very impressive.

What we learned

One person from our team knew ASL and one person did not know ASL. The team member who was working on the code aspect was the one who did not know ASL. From this project I have learned so much more about the ASL community and now can sign some words! I have learned that gemini AI is a great tool and you can accomplish a lot even if it is only one person coding with it.

What's next for tASL

Personally, I will be using it whenever I can because I want to be able to speak and understand ASL. I would love to keep working on this project. I would do all of the following if I would have free access to all of what Google Gemini can offer. I would add common words in the ASL directory so when the user inputs a word it knows then it gives them a gesture instead of having to spell everything out. I would like to make this like Duolingo where the user can learn new ASL everyday which would require the user to make a account. I would like to work with google and make this a language they offer on google translate.

Built With

• esm.sh
• gemini
• github
• mediadevices
• mediarecorder
• requestanimationframe
• speechrecognition
• tailwind
• typescript
• wikimedia