GestureAI

• 

  predicting multiple hands

• 

  predicting open palm

• 

  predicting closed palm

Inspiration

As big fans of video games, our ideas came from quite an unexpected place, online streamers. We saw these streamers playing games with the weirdest possible controls, i.e. beating games with a guitar, and thought how cool it would be to create something new ourselves and push the limits of what's possible.

What it does

Our website allows users to train their own machine learning models to recognize specific hands gestures. After binding them to keyboard or mouse inputs, the user can then control their computer using these custom gestures, completely hands-off. This opens up a ton of opportunities to both play game in a more unorthodox manner and perform more productive tasks such as controlling presentation slides at a distance.

How we built it

We utilized multiple python libraries for this projects, including:

• pyAutogui for keyboard & mouse control
• Tensorflow for machine learning/model creation
• Flask for backend web development
• Mediapipe for providing hand segmentation data

Challenges we ran into

It was also our first hackathon and first experience with tensorflow and machine learning. One of the biggest challenges that we've faced was porting the cli code into a web development framework.

Accomplishments that we're proud of

We're very proud of how smooth the experience of training and prediction is for the user. This was our first hackathon and we are all very proud of the progress we made, especially since we used so many new technologies.

What we learned

We've learned how to create tensorflow models using data from a different machine learning model.

What's next for GestureAI

The next step is to allow for binding gestures to a macro instead of just a single action.

Built With

• flask
• mediapipe
• pygame
• python
• tensorflow