ASL to Text and back (Team BananaDogs on discord)

Example Usage

Inspiration

We wanted to write an app that applied computer vision which could be helpful for real people. Communication online is incredibly important in recent years, and we want to make sure deaf people can communicate in ASL without alienating their peers. Sign to text aims to help.

What it does

The tool is able in intercept a video call giving you, the user, real-time translation of the signs given by the person you are conversing with. Then, when you wish to speak, it will translate text you write into a series of signs to show your teammate.

How we built it

We split into 3 parallel tasks:

Design and create the easy-to-use GUI
Make a text-to-ASL translator
Train a computer-vision model to detect signs in an image

These were mostly separate tasks until the final hours of the competition where we integrated them all together.

Challenges we ran into

We ran into issues with bias in our data making the computer-vision model preform poorly on certain skin tones. It turns out this was due to over-fitting on our training dataset. Re-training for less time produced a slightly less accurate model, but then is preformed better on all skin tones.

We also found that making the GUI cross-platform posed to be difficult because we were all on different operating systems. Lots of testing across platforms seemed to be the only way around this. It probably took up more time than it needed to.

Accomplishments that we're proud of

The app mostly works as an ASL training device. It doesn't work so well on a video call, but we've found that it has taught us quite a bit of ASL already!

What we learned

Cross-platform applications are hard, especially when you need special features like screen sharing.

And a higher accuracy in a machine learning model does not mean a better model. Be wary of over-fitting, it can easily turn a great model into something useless.