MelodyMotion

Turn your webcam into a virtual instrument by playing melodies with your fingers. It tracks finger gestures and maps them to musical notes, letting you compose music in real-time.

Comment

🎵 MelodyMotion

Turn your webcam into a musical instrument — compose melodies with just your hands!

💡 Inspiration

We wanted to create a fun and interactive web application that everyone can engage with—especially those who don't have access to real musical instruments. MelodyMotion allows users to virtually play instruments using only their fingers and a webcam.

⚙️ What It Does

MelodyMotion transforms your webcam feed into an interactive musical canvas:

  • Tracks finger movements in real-time using ml5.js.
  • Divides the screen into zones, like keys of a piano or strings of a guitar.
  • When a finger moves across zones, it plays the corresponding note or sound.
  • Users can choose how many fingers (1–5) they want to track.
  • Each detected fingertip plays a different note based on the zone it is in.
  • Generates a continuous melody as users move their fingers—like strumming or tapping an actual instrument.

🧱 How We Built It

  • ml5.js – Used for real-time hand and finger tracking.
  • Next.js – Built the frontend UI and managed state updates efficiently.
  • Web Audio API – For precise control over audio playback and effects.

🧗 Challenges We Ran Into

  • Handling real-time video input and gesture detection simultaneously.
  • Avoiding unnecessary React re-renders during rapid UI updates.
  • Ensuring accurate and smooth melody generation mapped to finger movement.

🏅 Accomplishments We're Proud Of

  • Successfully tracks 1 to 5 fingers with reliable real-time performance.
  • Built an interactive music creation system based on hand gestures.
  • Achieved consistent and responsive melody generation with minimal latency.

📘 What We Learned

  • How to integrate real-time gesture tracking using ml5.js in a web app.
  • Techniques for melody generation using code.
  • Strategies to optimize React for real-time interaction-heavy applications.

🚀 What's Next for MelodyMotion

  • Build an on-screen virtual keyboard or instrument layout for visual feedback.
  • Add recording functionality to save and replay melodies.
  • Support for multiple instruments with more realistic sound sets.
  • Allow custom sound mappings and user-defined gesture zones.
  • Introduce gesture-based effects (e.g., volume, pitch bend).
  • Make the app more accessible and customizable for all types of users.

🎶 Play music with motion — no instrument required!

Built With

  • ml5.js
  • next.js
  • tailwindcss
Share this project:

Updates