Pinch Draw

Inspiration

We’ve wanted to get into digital art because of the larger array of drawing tools we have access to. However, drawing on a phone can be less precise due to your thumb being a bigger part of the screen, drawing using a touchpad is difficult since you constantly have to apply pressure, and drawing tablets can cost hundreds of dollars. That led us to create Pinch Draw, an underwater themed air drawing game where players use hand gestures instead of traditional drawing tools. We were also inspired by Quick, Draw! and its fast, prompt-based drawing experience. We also added a competitive two-player mode because we thought it would be fun to turn sketching into a competition.

What it does

Pinch Draw is just a browser based air drawing experience powered by the webcam and hand tracking! In Solo Quick Draw, a player completes a pinch check, receives a random drawing prompt, and uses their thumb and index finger to paint on the digital sketchboard. In this mode, players can also change brush colors through hand gestures, adjust brush size, switch between bubble paint and smooth paint, clear their work, and save their drawing as a PNG. In Double Drawdown, two players share the webcam and draw the same prompt at the same time. To solve the issues of the interaction between both hands, the camera is divided into left and right interaction zones so each player controls only their own part of the canvas. And after the round, players can compare their doodles and vote for whose doodle is better.

How we built it

We designed Pinch Draw’s underwater themed interface in Figma, then built the app using HTML, CSS, JavaScript, Next.js, and TypeScript. For hand tracking, we used MediaPipe’s Hand Landmarker model through @mediapipe/tasks-vision. The app accesses the user’s webcam with the browser’s getUserMedia() API, detects up to two hands in real time, and uses thumb and index-finger landmarks to recognize pinch gestures. We also rendered the drawing experience using the HTML Canvas API. The hand landmark coordinates were mirrored and mapped onto the canvas so users can draw naturally by moving their pinched fingers in front of the webcam. To improve the experience, we added landmark smoothing, separate pinch on and pinch off thresholds, and a short release grace period to reduce jitter and accidental interruptions while drawing. For Solo Quick Draw, users complete a pinch check before receiving a random ocean themed prompt and a timed drawing round. While drawing they can control brush size controls, bubble and smooth paint styles, canvas clearing, and key-free color selection with recognized hand gestures. They can also save their drawing as a PNG. For Double Drawdown, we used MediaPipe’s ability to track two hands and divided the mirrored webcam feed into left and right interaction zones. Each player’s hand is assigned to its own drawing board, allowing two people to draw simultaneously using a single shared webcam. At the end of each timed round, the players compare their drawings and select a winner.

Challenges we ran into

One of the biggest challenges was making the hand tracking sensitivity match the pace of the hand movement. The webcam landmarks naturally jitter, especially when the hand moves quickly or the lighting changes (ex: the glare from the window/bulbs). So, we had to smooth the hand positions while still keeping the cursor responsive enough to feel like drawing in real time. Another challenge was the pinch detection. The pinch couldn’t be treated as a single exact distance because people hold their hands differently and camera tracking is not perfectly consistent. So we used these separate pinch on and pinch off thresholds, along with a short release grace period, to prevent drawing from rapidly turning on and off. The two player mode also introduced another challenge. Both players needed to share the same webcam without controlling each other's boards. We solved this by dividing the mirrored camera feed into left and right zones and mapping each detected hand to the correct player canvas. And finally, gesture based color selection was difficult because hand signs can look different depending on the angle, distance from the camera, and which hand is being used. To add on to these limitations, the lighting and the glares from the ceiling lights and windows made it even harder for the webcam to detect the finger action/gesture.

Accomplishments that we're proud of:

Creating a smooth pinch to draw system with a live webcam cursor. Building an underwater themed interface, by learning figma from ground 0. Adding hands free color selection through gesture recognition, and an option to choose the color through cursor if the gesture gets blocked by any light. Creating both the Solo and Double modes. Separating two players into left and right webcam zones so they can draw simultaneously using one camera. And most importantly, learning languages, frameworks, and designing websites we previously had no idea about!

What we learned

Through this project, we not only learned frameworks and designings but also how much iteration is needed to make computer vision interactions feel natural. For example, a gesture that works once in perfect lighting may not work reliably in real world conditions, so thresholds, smoothing, state management, and user feedback are essential. We also learned how important visual feedback is in gesture controlled experiences. Features such as cursor ring, pinch indicators, green drawing state, color previews, and status messages help, not only the users, but the developers too to understand what they system sees and what action is currently active. Finally we learned how to design around the limitation of a shared webcam. Building a two player experience required us to think carefully about mirrored coordinates, screen regions, independent canvases, and preventing one player’s hand from interfering with the other player’s drawing area.

What's next

In the future, we hope to identify a software that would help us block out natural light so that hand gestures can be read even through glares. We also hope to train an AI model (potentially Vision Transformer) that can judge how good the art is by declaring the winner by itself in the double mode without having the players vote themselves. And for the single player mode, we hope that it can provide a score out of 100.