TD American Sign Language Helper

To foster an inclusive future, we used the Clarifai and TD Da-Vinci API's to create an ASL to English translator.

Comment

Inspiration

Social purpose is paramount in this era. As part of our commitment to creating a more inclusive future, we worked with the Clarifai and TD Da-Vinci API's to create a translator from American Sign Language to English text, enabling those who have speaking disabilities to be able to easily communicate with their computer and with others.

What it does

We begin by having the user take a picture of the letter they would like to use in sign language – we have letters a, b, and c as options for users to interact with their TD banking accounts. Once the user has taken the picture, this picture is then analyzed to determine the text translation, which is used to dictate the command for the Da Vinci API. The user is returned some form of banking information based on their selected command.

How we built it

We used Clarifai's API and trained it on this dataset so that it could recognize ASL. Then we used this trained model to interpret different options and this, in turn, sends the predicted output to the TD API to get the information of the user bank account details.

Challenges we ran into

Sending data from frontend to backend, limited training set size for Clarifai API calls, Understanding Node js promises

Accomplishments that we're proud of

We are proud that we were able to implement the prototype from a raw idea.

What we learned

We got an in-depth knowledge about using the API's we leveraged. Also, using Promises in Nodejs was a challenge which we loved to overcome.

What's next for TD American Sign Language Helper

While limited time to build our product was one of the biggest constraints we faced, there are some things we have planned for the future of this product’s success, not only for TD, but to provide better communication and interaction for those who are disabled. We are looking forward to integrating facial recognition for the identification of the User, which will make processes faster for our users. Additionally, due to constraints with the Clarifai API version we were provided, we had to limit ourselves to implementing just three endpoints for the TD Da-Vinci API's. In the future, we can expand, and take full advantage of the Da-Vinci’s features.

Share this project:

Updates