GestureLab — Touchless Creative Studio

Turn hand movements into music, light, and digital art. GestureLab uses real-time computer vision to create an interactive, touchless creative experience.

Comment

Inspiration

What if we could create music, change colors, and draw without touching a keyboard, mouse, or screen?

This question inspired GestureLab 2.0, a browser-based creative studio that transforms hand movements into interactive digital experiences.

I wanted to explore how computer vision could make everyday technology feel more natural, playful, and intuitive. Instead of relying entirely on traditional controls, GestureLab lets users interact with creative tools through simple hand gestures.

What It Does

GestureLab 2.0 combines three interactive experiences in one web application:

  • Gesture Music: Create and explore sounds by moving your hand. Hand position influences musical notes and sound characteristics.
  • Gesture Lights: Change colors and brightness through hand movements.
  • Gesture Draw: Use your finger as a virtual brush and draw directly on a digital canvas.

The application uses a webcam to detect hand movements in real time. An index finger acts as a virtual cursor, while a pinch gesture allows users to interact with controls and draw.

Users can also explore the application using a mouse or touchscreen without enabling the camera.

How I Built It

GestureLab 2.0 was developed using HTML, CSS, JavaScript, and Google's MediaPipe Hand Landmarker.

MediaPipe detects hand landmarks from webcam input. JavaScript processes the detected coordinates and gestures to control interactive elements.

The application uses:

  • MediaPipe Tasks Vision for real-time hand tracking.
  • JavaScript and the Web Audio API for interactive sound generation.
  • HTML Canvas for drawing and visual effects.
  • HTML and CSS for the user interface and responsive layout.
  • GitHub and GitHub Pages for source-code hosting and public deployment.

AI-assisted development tools helped me build and refine the prototype. My role included defining the project concept, guiding its features, testing the interactions, identifying problems, and working through improvements.

Challenges I Faced

One of the biggest challenges was integrating real-time hand tracking into a browser-based application.

During development, I encountered a dependency issue involving the MediaPipe package version. Resolving it required using a compatible release and testing the application again.

Another challenge was making the experience practical and interactive rather than simply displaying hand-tracking data. The goal was to connect finger movement and pinch gestures to creative actions that users could immediately understand.

Deploying the application also presented difficulties. Initially, the published website loaded without its intended styling because required project files had not been uploaded. After adding the missing CSS and JavaScript files, the public version displayed and worked correctly.

Accomplishments

I am proud that GestureLab 2.0 has become a working interactive prototype rather than just a concept.

The completed application supports webcam hand detection, finger tracking, pinch interactions, music controls, interactive lighting, and digital drawing.

It is publicly accessible through GitHub Pages and can be demonstrated directly in a web browser.

What I Learned

This project helped me understand how computer vision can connect physical movement with digital interaction.

I learned more about hand landmarks, gesture-based interfaces, browser APIs, dependency management, debugging, and web deployment.

I also discovered how important testing and iteration are when turning an idea into a functional experience.

Most importantly, I learned that a relatively simple concept can become an engaging interactive product when technology and creativity work together.

What's Next

I would like to continue developing GestureLab by:

  • Improving gesture recognition and interaction stability.
  • Adding more creative tools and customizable gestures.
  • Exploring accessibility-oriented use cases through future user testing.
  • Expanding the drawing and music features.
  • Making the experience more adaptable across devices.

My long-term vision is to explore how gesture-based interaction can make digital creativity more expressive, intuitive, and enjoyable.

Built With

Share this project:

Updates

Submission history