Handcut

After assign action to handgesture, in real time it will perform equivalent computer actions for each gesture upon detected
Here is our data sample
This is our user interface used to change the action being performed for each hand gesture
App used to collect data
Model training

Inspiration

I often do video editing, 3D modelling and programming or computer works in general. These works require me to know a bunch of computer actions that can be accessed via mouse click or computer shortcuts, mouse click is slow and i find computer shortcuts to often be hard to remember and click. Therefore, I ask myself why not use handgestures and actions to control computer.

What it does

Detect real time hand gestures and actions using machine learning to perform computer actions. This would help to improve efficiency and productivity

How we built it

I build it using mediapipe to collect landmarks data of hand gestures then use tensorflow and sklearn to train the data with LSTM layers (there are 35 video samples for each gesture and each video has 12 frames). Then, use the model to predict real time data of hand gestures/actions and perform equivalent computer actions.

Challenges we ran into

There are tons of challenges that we ran into. First of all, I don't have access to any pre-trained model or database so I have to create an appliation to collect data then collect all the data myself. Most people in our teammates get problem with tensorflow and opencv so there is only one person can do the training. We have to be quick so we can only do 35 video samples for each gesture, each video is 12 frame. With such a small data size, and not enough time to figure what model or layers fit best, we also get lots of challenges in creating the model. However, at the end it turns out to run pretty smooth. Our biggest problem is time, after collecting the samples and train the data, we barely have any time left to finish the application, so at the end it is very rush and we can't make a good demo to reveal how powerful our application is.

Accomplishments that we're proud of

The model works and is pretty accurate at detecting with 8 successfully trained gesture (this is the first time that I train a model from nothing to being running successful at this such short time interval). With the trained gestures, we can perform any computer actions with it using binding shortcuts. This gesture control can help people become more efficient and productive

What we learned

This is my first ever official machine learning project that everything is built by myself, from making application to collect data, to collecting data, training data, frontend, backend to perform actions in just 36 hours. So I learn the basic concept of making a machine learning model and apply it tor real life situation

What's next for Handcut

Handcut is a very powerful tool, it can perform basically any computer actions with handgesture/action as long as there is a shortcut key binding to it. So for video editer, designers, programmer or basically everyone, they can assign the video editing, designing, programming or any daily comptuer action tools to their hand gestures/action and perform it much productively. Besides that, this can be used as a sign language reader to help the deaf or hard-hearing people communicate quicker and more efficiently.