Gesturing App

Command the canvas with your hands!

Comment

Inspiration

My hand-controllable drawing (A.K.A. gesture-ing) app is inspired by holograms in superhero media (specifically with the show I've been watching, Agents of Shield). With a wave of your hand, the 3D display could switch to another screen, or with an imitation of a press of a button, the hologram could simulate interactivity. I'd imagine how fun it is to manipulate technology with just your hand gestures. And when I saw an example idea listed under the theme on hackthearts.net, "A drawing tool controlled entirely by gestures", I knew I had to do it. So I made a drawing app that users could interact with with their hand gestures.

What it does

This Gesturing App allows the user to draw onto the canvas basic shapes in respect to specific hand signs, and allows them to move the shape around by moving their hands around (and to "stamp" the shape permanently by closing their fists). For example, the "peace sign" would correspond with the shape" triangle", and just raising your index finger ("one sign"), would correspond with the shape "vertical line". The shapes can vary in size and color using keyboard inputs. Once the user is satisfied with their canvas, they could press 'S' on their keyboard to save that image to their computer.

How I built it

I made the project and used libraries in Python. I first used the library "open-cv2" to get frame inputs from the camera. After some image processing, I then feed those frames into the "Google MediaPipe Hand Detection" system in order to determine if there is a hand in frame, and if yes, the (relative) locations of the hand's joints (what's called "hand landmarks" in the MediaPipe documentation). This allowed me to use each hand landmarks' position and rotation (of the finger's tip and its lower joints) to determine whether a finger is held up, thus letting me know what gesture the entire hand is doing. The gesture is then translated into a respective shape (that I custom made using the drawing tool "PiskelApp"), which I draw on the canvas, which is created using the Surface object in the "Pygame" library. And once I determine the hand sign and shape, I can move on to allowing the user to move the shape on the canvas around, which I did by tracking the location of the entire hand on the camera image every frame, and comparing it to the last frame's, to determine the offset, which I then apply (relative to the canvas' dimensions) to the drawn shape on the canvas. I also wanted a way for the user to see their own camera output, so I added that as well (along with instructions on the top-right) with Pygame.

Challenges I ran into

One of the first problems I ran into was with the camera output. The opencv library returns images from the camera flipped around the y-axis and in Blue-Green-Red format, while the hand detection system expects upright and RGB-formatted images, and at the same time Pygame expects rotated-90-degrees-clockwise, flipped-only-around-the-x-axis ones. The conversion of image formats between 3 libraries was something I took a lot of time figuring out on. The second most time-consuming problem was dealing with fixing the shape's position when the user's hand goes out of frame and comes back. Originally, I thought the shape would retain its previous position when the hand comes back into the camera and allow the user to start moving the shape again where they left off as a byproduct of my code, but when that was demonstrated untrue, I retreated to hard-coding a "gesture_lost" variable, which does work but requires a bit more of it. Another more repetitive problem was the pygame GUI. The positioning on the title and informations was something I messed around with for quite a while because they had to be in absolute pixel positions, not relative ones.

Accomplishments that I'm proud of

I was able to integrate 3 libraries at once (opencv, mediapipe, pygame) into one cohesive application, without quitting towards the end (which is something I often do). I am also very proud of myself for reading the documentation, specifically how pygame surfaces and mediapipe's hand landmarks work. The customization of shapes is also another thing I'm really proud on. The features of colorization and scaling of the shapes is not something I planned on doing in the first stages of development, but I implemented it rather efficiently, I'd say, in the end; and in my opinion, really elevated the interactive experience of my whole drawing app.

What I learned

I learned how to get camera input, how to detect hand gestures using MediaPipe, how to draw surfaces in Pygame, and I learned how to alter an image's color at runtime (which I used for the shapes).

What's next for Gesturing App

I would like to add rotational support for gestures. Currently, the system only detects hand gestures in their upright position; if I could update it to add rotational detection, I could also then add rotations to the shape, which would propel this app into a more legitimate drawing app. Next, a more obvious one, I would like to add support to even more gestures and shapes. Perhaps a thumbs up draws a star shape, or perhaps the telephone sign could draw an arc. The current system of saving, undoing, scaling, color-choosing, and centering using keyboard input is something I want to replace to be entirely hand-controlled too.

Built With

Share this project:

Updates