Jesture

Draw to your heart's content!
Challenge your friends!
Use colorful tools!
Challenge your friends!

Inspiration

The inspiration for Jesture mainly comes from popular fiction. The idea started as one one-off comment by a couple of my friends who thought it would be interesting to be able to control your computer through hand gestures like in Iron Man, where Tony Stark uses hand gestures to control devices or interact with virtual environments. This concept felt futuristic and exciting, but we also saw a practical application: making technology more accessible and intuitive. Gesture-based controls can provide an alternative interface for individuals with limited mobility or those who find traditional input devices like keyboards and mice challenging to use. By combining creativity and accessibility, we aimed to create a tool that is not only fun but also empowers users to interact with technology in a more natural and inclusive way.

A paint application is the perfect demo for this new idea as it showcases the potential of gesture-based controls in a highly interactive yet fun way. Drawing is a universal activity that doesn't require any prior technical knowledge, making it accessible to users of all skill levels (even kids, in this case). By allowing users to create art with simple hand movements, the application demonstrates how natural gestures can replace traditional input devices like a mouse or stylus. A paint application also highlights the precision and responsiveness of the gesture recognition system. "The artists" can see immediate feedback as their gestures translate into lines, shapes, and colors on the canvas, making the technology feel seamless and engaging.

Generally, a paint program provides a solid base to explore features, whether it be the general controls of a gesture-based system or integrating Gen-AI to guess the user's creations.

What it does

Jesture is an interactive drawing and guessing game powered by gesture recognition. Users can draw on a virtual canvas using hand gestures, switch between drawing modes (free draw, line, circle), and have an AI guess what they are drawing. It combines creativity, technology, and fun into one seamless package.

How we built it

MediaPipe for real-time hand gesture recognition.
Raylib for rendering the drawing canvas and UI.
OpenCV for capturing webcam input.
Google GenAI for generating AI-based guesses of user drawings.
Python as the primary programming language to integrate all these technologies.
Tensorflow used to train the hand gesture recognition (thumbs up/thumbs down/pointing/etc.)

Challenges we ran into

Fine-tuning gesture recognition model to ensure accurate and responsive controls.
Managing real-time performance while processing webcam input, rendering graphics, and running AI predictions simultaneously. (Some of my earlier implementations were painfully slow)

Accomplishments that we're proud of

Successfully integrating gesture recognition with a drawing application.
Creating an "objectively" fun and interactive experience where AI guesses user drawings.
Building a smooth and responsive interface that feels intuitive to use.

What we learned

How to work with MediaPipe for gesture recognition and its potential for other applications.
The power of Raylib for creating interactive graphics in Python.
The importance of optimizing performance when working with real-time systems.
How to integrate AI models like Google GenAI into creative projects in a fun way