sli.ai

Inspiration

We wanted to simplify communication between any user and a person who speaks mainly sign language.

What it does

In one direction, it converts sign language from camera input into text to be displayed to the user. On the other hand, it takes speech from the user and coverts it into text to be displayed to the person who mainly speaks sign language.

How we built it

The entire front-end was built using VueJS, with speech recognition being done using Chrome's Web Speech API. The two machine learning models were built. The first model is a frozen pre-trained model which works with a convolutional neural network. The second model is one that was built used Microsoft's custom vision, with manual images being taken and fed.

Challenges we ran into

Getting a model that works well for detecting sign language.

Accomplishments that we're proud of

Getting a semi-working model for detecting sign language.

What we learned

Loads of machine learning knowledge.

What's next for sli.ai

Supporting displaying to multiple screens at once. Refining the machine learning model to be more accurate. Implementing text-to-speech for sign language that's converted into text via the model.