Sign language gives a voice to deaf people - the only problem is that so few people know how to use it. In the modern day, sign language is, in effect, only handy for deaf people when they are in the presence of other deaf people, because one can safely assume that most people who are not deaf, and who have not experienced an explicit need to learn sign language, have not learned it - it is simply too much of a hassle. Few things evidence this more than the recent debacle that took place during the evacuation of Florida in the presence of Hurricane Urma, when sign-language-"fluent" officials signed to Florida's deaf population to beware of "pizza" and "bear monsters." Clearly, learning sign language takes too much effort and offers too little reward.

The Helping Hand remedies this problem by making sign language easy, and a pleasure to learn. The pleasant interface keeps the user hooked, while the interactive process ensures the user gains knowledge at a steady pace and constantly reviews everything he or she has learned. Learning sign language has gone from an endeavor requiring several days to one requiring a single relaxed afternoon.

What it does

The Helping Hand is an interactive tool for learning about sign language. Users can take advantage of its simple interface to quickly learn the major signs of ASL, and can then practice their skills using the tools available on the website, including several computer-vision based tools, built using machine learning techniques, that use the user's webcam to detect signs made by the user and determine their correctness.

How we built it

We built the Helping Hand's interface using React.js and Materialize, and we used these to take pictures client-side and to send them to the server-side. Python largely handles the back-end, controlling what happens to these images and data, and containing computer vision and machine learning scripts that process this data. Python then returns this data to JS for presentation to the user.

Challenges we ran into

Our website was large, with a lot of planned features, and we were heavily constrained by time restrictions. Also, some of our project's server-side dependencies were rather large and could not fit into the very limited file size allotted for file entries.

Accomplishments that we're proud of

We're proud that we were able to integrate machine to make our design considerably more accurate than it would have been without a rigorously computed CV algorithm.

What we learned

We found out that Python Bottle does not work very well in conjunction with ReactJS - this is a combination we will seek to avoid in the future.

What's next for The Helping Hand

The Helping Hand is useful - but it could be considerably more useful if it could reach a wider audience. A logical next step would be to turn the Helping Hand into an app - this would allow it to reach many more people, thanks to the App Store, than it would by simply sitting somewhere in a remote corner of the internet.

Share this project: