SignMeUp - The ASL Subtitle

Inspiration

My ASL teacher grew up navigating a world built for hearing people, constantly misunderstood because English was never fully accessible to him. Like millions of deaf Americans, his first language was ASL, not English; yet, society consistently judged his intelligence when it came to mastering English. His story inspired us to create SignMeUp because subtitles aren't just captions: SignMeUp is a solution for equity, access, and full participation.

What it does

The user can upload any MP4, and then our AI extracts the audio. After, it utilizes Whisper to transcribe the audio into text and uses our custom EnglishToASLConverter class to translate the text into ASL. Then we extract words from the ASL string and check if it's in the database, and then return the image corresponding to it, overlaying it on the video. Finally, we return the video and the overlay together.

How we built it

Using a Flask app, we allow the user to upload an MP4 file. Then the program extracts the file’s audio and uses OpenAI’s Whisper model to transcribe the audio to English text. After, we pass the said text to our EnglishToASL translation pipeline, which utilizes Stanford University’s Stanza NLP package for accurate translation, and then it produces a combined string of all the words we need to sign. We then pass each word in the string through the ASL-LEX database and return a corresponding image of the sign. Finally, we overlay the images over the video for the user to see!

Challenges we ran into

One of our main challenges was trying to find a data set of ASL words that the robot could perform. The best data set we found only had 2,500 words out of the 10,000+ words that are actually in the ASL alphabet. This only emphasizes the inaccessibility and struggles that deaf people still have to face.

Accomplishments that we're proud of

Throughout this process, we learned and practiced with a lot of new tools. While we were building the UI, we utilized animations for each part, a first-time effort for us. Additionally, we practiced and learned the nuances of 3D models when it came to displaying the signs from the data set. Lastly, the most gratifying part was learning and creating one coherent streamline for the full stack, AI, and the video overlay.

What we learned

Throughout this process, we learned how to use CSS animations. We also encountered the lack of data for helping those who are hard of hearing, as there were minimal ASL data sets, further emphasizing the inequality of resources in society today. We also learned how to create 3d models and 3d animations with Blender and how to utilize large datasets.

What's next for SignMeUp - The ASL Subtitle

After developing the program, we hope first to create it as a Chrome extension. This way, users can turn it on in their navigation bar and start watching all their videos. In the future, we hope to also pitch our project to primary streaming services such as Netflix or Prime Video, and become one of the accessible language options on their platform.