Inspiration

We started with a question: would you rather conduct an orchestra or catch a ball virtually? Inspired by interactive hand-tracking projects, we chose conducting as a creative way to make the technology tangible.

What it does

Music Grid tracks both hands with a camera. Move your hands to control the volume of the song’s vocal, drum, bass, and string stems, follow timed cues, and earn points for matching them. The music is made from AI-generated stems.

How we built it

We used Figma to explore the visual design, ChatGPT to develop ideas, and Codex to build the prototype. The app uses React and TypeScript, Google MediaPipe Hand Landmarker for hand tracking, and the Web Audio API to control the music.

Challenges we ran into

Getting both hands tracked reliably and mapping their positions to smooth, synchronized audio controls took careful iteration. We also considered privacy: the camera requires permission, and hand tracking runs locally without uploading or recording camera footage.

What we’re proud of

We built a runnable prototype where users can conduct the music with real hand movements. Timed cues, changing instrument levels, and scoring make the interaction playable.

What we learned

We learned to plan around an early prototype, connect specialized APIs to a real user experience, and use AI tools to support design and development while refining the idea ourselves.

What’s next for Music Grid

We want to explore the same interaction framework for virtual ball-catching, recreation and wellness activities, and learning experiences for children and older adults. Those are future ideas; Music Grid is the project we’ve built so far.

Built with

Figma, ChatGPT, React, TypeScript, Vite, Google MediaPipe Hand Landmarker, Web Audio API, and AI-generated music stems.

Built With

Share this project:

Updates

Submission history