Many in the deaf community prefer to watch someone signing instead of reading large portions of text in English. This makes sense, as ASL (American Sign Language) is their first language, and English may be their second. In cases where a deaf person was born outside of the United States, they may know ASL, but not English. In cases such as this, ASL signing is a necessity to communicate. However, most deaf people do have an interpreter with them at all times.
What it does
We built a chrome extension and Android App that allows a deaf user to highlight a block of text on a webpage, and translate it into ASL. The translation is done one word at a time, using video clips of an ASL interpreter who is signing those individual words.
How we built it
A backend server, running ruby-on-rails, accepts HTTP requests from a client. The request should contain the text that is to be translated. The server parses the text, one word at a time, and looks them up on the ASL dictionary site handspeak.com. The videos are pulled from handspeak.com and stitched together, and returned to the client as a single video.
The client can be anything that can send HTTP requests, and can play MP4 videos. For the hackathon, we create a chrome extension that allows a user to highlight any text on a webpage, right click it, and then click a "Sign with Handy" menu item. This will open a new web page with the generated video of the signed text. The Android App we developed works in a similar fashion. You can select any text from a web page or another app, click the share button, and then select "Handy". The Handy app will open and display the generated video of the signed text.
Challenges we ran into
We had challenges fetching videos from the handspeak.com. We also had challenges integrating the client(s) with the server.
Accomplishments that we're proud of
Laying a foundation for an accessibility service that allows those whose first language is ASL, to translate any text into ASL automatically and without the need for an in-person interpreter.
What we learned
We learned about the (many) differences between English and ASL, and the challenges involved in translating from one to the other.
What's next for Handy
1) compensate for vastly different grammatical/sentence structure between English and ASL. 2) compensate for emotion (i.e. exclamation points). 3) compensate for regional ASL differences and dialects
Jen Ward Janice