Handle

Inspiration

We wanted to try a computer vision project that would be challenging for both of us and to learn new skills.

What it does

The program takes frames from the built in camera on the computer, isolates skin from the rest of the image and sends the image to a neural net that we trained with data we created on saturday night of different hand symbols, which then tries to determine which symbol it is shown.

How we built it

We used opencv library to develop the video capture side of the program, using some research papers and other available data to isolate skin. We then used tensorflow to retrain a convolutional neural net to identify the ASL symbols a,e,i,o,u and the common hand signs for peace, heart, and thumbs up.

Challenges we ran into

Getting the camera to isolate skin was more of a challenge than we expected, and we ended up doing more research than we initially anticipated to really understand how to make it work.

Another big challenge was data collection. Due to limited time, we trained each symbol with 400 images taken with two people's hands, and found the data was not nearly random enough to make a robust set. In the future, we will try to get more varied images to make our program work better.

Accomplishments that we're proud of

Really understanding what opencv is doing each time we used it, figuring out the skin isolation, and retraining a neural net.

Interpreting the output of the neural net processing portion and applying that information to our UI.

What we learned

Taking the time to read can save a lot of time downstream.

Good data goes a long way

What's next for Handle

We'd like to improve our imaging program to get crisper images of hands. From there, we'd like to train a neural net to recognize the whole ASL alphabet with much more varied and robust data.

An interesting test of this program in the future would be to try to close caption some simple videos using our program.

Built With

• opencv
• python
• tensorflow