SLaM: Sign Language Model

Inspiration

Contrary to popular belief, a lot of deaf people know sign language, some prefer speaking or using hearing aids to communicate.

It would also be very daunting to learn a new language, and expensive, since American Sign Language doesn't necessarily follow Subject-Object-Verb.

This could be unintuitive and daunting for adults to learn.

With our project, we hope to increase competency in American Sign Language (ASL) by gamifying it and hammering the regular English way of grammar with repetition of sentence structure that would be possible.

We use computer vision and pose estimation to detect the sign language of the person and help them improve.

What it does

We did not finish it, our website is barebones, and the AI model to detect sign language is not complete.

How we built it

Vivian and Tallulah used Figma and it's dev mode to build a website.

Challenges we ran into

We've never used Figma and React before so there was a major learning curved for that. We spent a lot of time trying to convert our Figma design into a useable website.

There was a lack of documentation on mediapipe, and it seems to be deprecated for local use, and Google wants people to use media pipe through their GCP's Gemini API.

Also normalization of video frames since every video have a non regular number of frames, so accuracy might not be great.

Accomplishments that we're proud of

Talulah and Vivian: We are very proud to have been able to create a Figma design with some links. We also learned a lot on how to build a webpage and how react can be used. We made a button and we able to convert some of the Figma page to a webpage. For the sign language aspect we were able to get a working camera.

Zoella: I'm happy that I've able to build something that can get key points from 2000+ ASL words with 50k+ short videos.