Inspiration
This project was inspired by the idea of making human-computer interaction more natural and intuitive. Instead of relying on traditional input devices like a mouse or touchscreen, I wanted to explore how simple hand gestures could control digital experiences. The concept of an “Air Canvas” and gesture-based gaming felt like a perfect way to combine creativity, interactivity, and computer vision into something engaging and futuristic.
What I Learned
While building this project, I learned a lot about real-time computer vision and user interaction design. Working with MediaPipe Hands helped me understand how hand landmarks are detected and how to translate those into meaningful gestures. I also learned the importance of smoothing, debouncing, and stability in gesture-based systems—small fluctuations can completely break the experience if not handled properly. On the frontend side, I improved my skills in managing UI states, animations, and designing intuitive user experiences.
How I Built It
The project is built entirely in the browser using JavaScript, HTML, and CSS. I used MediaPipe Hands for real-time hand tracking and the HTML5 Canvas API for rendering both the drawing interface and the game.
I first implemented the drawing system, where the index finger acts as a pointer to draw on the screen. After stabilizing gesture detection, I extended the project into a game mode where users can aim and shoot birds using hand gestures. The application is structured into two modes—Draw Mode and Game Mode—with a shared camera and detection pipeline. I also focused on UI/UX improvements like mode switching, instruction overlays, and visual feedback to make the experience smooth and user-friendly.
Challenges I Faced
One of the biggest challenges was making gesture detection reliable. Initially, small unintended movements or slight finger misdetections caused incorrect actions like accidental clearing or mode switching. Solving this required implementing smoothing techniques, frame consistency checks, and time-based gesture validation.
Another challenge was handling lighting and camera input—too much brightness affected detection accuracy, so I had to adjust the video feed dynamically.
On the game side, designing a balanced and engaging experience (like bird movement, collision detection, and timing) while keeping performance smooth was also tricky. Ensuring that new features didn’t break existing functionality required careful modular design and testing.
Overall, this project helped me understand how to bridge computer vision with real-time interactive applications, and how important usability and stability are when building gesture-based systems.
Log in or sign up for Devpost to join the conversation.