While the deaf and hard-of-hearing population can visually communicate with people educated in ASL, video chat can enable a new layer of communication. Translating from ASL to text streamlines that communication and makes Agora more accessible.

What it does

Intent: When two Agora users video chat, a bounding box around one area of the video stream captures hand gestures and classifies them using an SVM multi-class classifier on the ASL alphabet. The corresponding text appears as messages on the other user's screen. What actually happens: The ASL works separately from the video chat component. Still working on putting the two together.

How we built it

We found and tweaked an SVM classifier, and trained on a set of 2000+ images. We also found and tweaked an agora webapp. The two pieces are coming together now, but still in progress.

Challenges we ran into

We originally intended to provide ASL-to-action translation using the Amazon Alexa. We ran into issues since the only input stream to trigger Alexa actions is voice, which would make our service moot for deaf users.

Accomplishments that we're proud of

We got the ASL to text component working successfully and learned a lot about SVMs and opencv in the process.

What we learned

  • Opencv is really cool
  • Multi-class classifiers get worse if you add too many classes
  • Planning out all the parts of a project is important before getting started
  • Not everything works out the way you think it will

What's next for ASLink

We'll be iterating on this project at the next hackathon to see where it leads. We'd like to work more on the ASL to action translation using other IOT devices.

Share this project: