Air Drawing

The AirDraw controls demo webapp

Inspiration

We were both very interested in computer vision and its applications, so we wanted to introduce ourselves to it through a fun ML project.

What it does

The project will be a program that detects hand signs via a camera, with which the user will be able to draw shapes. Each sign will correspond to a basic shape (oval, rectangle, triangle, line). We also have a delete and move function that would be used in the actual game to delete the previous shape and move the current shape respectfully.

How we built it

The dataset that was used for the hand sign detection was ASL Alphabet | Kaggle which consists of a training set containing 3000 images for each category (ASL letter) and testing set containing one image for each category. Only a few letters were used, namely: L, A, S, G, K, V, B, and R. After preprocessing the data, we used a random forest classifier model which yielded very good results. We then implemented the model into a live video feed and deployed it in a web app using streamlit.

Challenges we ran into

At first, the model behaved very poorly when testing the model with a live video feed, and was working reliably with the left hand, but very poorly with the right hand. The issue was resolved by flipping the image horizontally when a right hand was detected. Furthermore, implementing the Model into a web app was challenging because we were originally planning on using flask, but we could not find a way to put the video feed. However, we managed to find a really cool library called streamlit that allowed us to deploy our model easily.

Accomplishments that we're proud of

Our model performed really well with a training accuracy of 0.8992 and a testing accuracy of 0.9017, our web app also works very well and is visually pleasing.