Handi

Handi is an online tool that utilizes the TensorFlow Object Detection API to create a transfer learning model to help teach and recognize ASL.

Comment

Inspiration

We wanted to work on a machine learning project in a group setting because we finally get an opportunity to work in different areas but bring it together at the end for an amazing project. The idea of identifying sign language seemed super neat to us and we wanted to learn it.

What it does

It utilizes a set of data that we made utilizing python and the open source program LabelImg to label the training data. From there it was built off the TensorFlow Objection Detection prebuilty model given via their API. It is able to detect hand signs and label them on video.

How we built it

We built the web app using ReactJS, and utilized Jupyter Notebook for creating the machine learning model. We took references from YouTube for learning how to develop the model. The model was originally in python, so it was converted into JSON and hosted on Google Cloud Object Storage to be used on the server side, which was required by React.

Challenges we ran into

The biggest challenge was using Google Cloud. To properly fetch our model, we needed to set up CORS configuration. Unfortunately, there was a time delay between us updating it and Google officially updating it on the server side (1 to 2 hours). This was not know until extensive research was done. Asides from that, dependency issues and outdated TensorFlow code made it extremely hard to debug, and we had to work around a lot of packages by moving code around within the packages. We also were short two members, so the amount of work we could've done was very limited.

Accomplishments that we're proud of

We are very proud of the front end we made and the machine learning model. It was a great accomplishment knowing that the images were trained off of our own images.

What we learned

We learned what it took to set up the dependencies required to train and export a TensorFlow model. In addition, we also learned how to connect those two models. One of the hardest hurdles in this project was learning how to work with python dependencies. Working to maintain environments was also very challending.

What's next for Handi

We hope to further gamify Handi and the learning process by adding a point system to the quiz. In addition, we want to improve the flashcards, which was something we could've gotten to if we had more time.

Share this project:

Updates