nodal

a musical playground: sculpting music with your hands

Comment

Nodal

Inspiration

We wanted to bridge the gap between physical gesture and digital music creation. Traditional MIDI controllers are limiting - what if you could conduct music with natural hand movements, seeing your musical ideas bloom in real-time as living, breathing visual structures?

What it does

Nodal is an interactive music performance tool that transforms hand gestures into MIDI notes with real-time visual feedback. Users can:

  • Conduct with hands or markers - Move two markers (or use hand tracking) across a multi-lane canvas
  • Generate polyphonic melodies - Control multiple independent voices simultaneously
  • See music grow - Watch notes spawn as animated nodal tree structures that evolve with your performance
  • Shape sound spatially - X-axis controls stereo panning, Y-axis controls pitch
  • Stay in key - Automatic scale quantization (Major, Minor, Dorian, Lydian, Mixolydian, Chromatic)
  • Send MIDI output - Connects to your DAW (Ableton, Logic, etc.) via virtual MIDI buses

How we built it

Tech Stack:

  • React with TypeScript for the UI framework
  • Web Audio API for audio analysis and preview tones
  • Web MIDI API for DAW integration
  • Canvas API for high-performance visual rendering
  • MediaPipe Hands (via HandTracker component) for computer vision hand tracking
  • Custom algorithms for scale quantization and directional lane selection

Challenges we ran into

  1. Hand tracking coordinate mapping - Converting webcam coordinates to canvas space while accounting for mirror-flipped video was tricky
  2. Multi-lane MIDI routing - Ensuring each lane sends to the correct MIDI channel without cross-talk
  3. Performance optimization - Rendering multiple canvas animations at 60fps while processing MIDI and audio analysis
  4. Touch vs. hand tracking UX - Making both interaction modes feel natural and responsive
  5. Scale quantization - Snapping pitches to musical scales while maintaining smooth gesture control
  6. Visual feedback timing - Synchronizing highlight animations, node spawning, and MIDI output without lag

Accomplishments that we're proud of

  • Seamless hand tracking integration - Natural, latency-free performance with computer vision
  • Beautiful generative visuals - The nodal trees create genuinely artistic, unique patterns for every performance
  • Professional audio routing - Rock-solid MIDI implementation that works with industry DAWs
  • Accessibility - Dark mode, multiple control schemes, and visual feedback for audio
  • Musical intelligence - Scale quantization keeps performances sounding musical even with imperfect gestures
  • Polish - Smooth animations, responsive controls, and thoughtful UI design throughout

What's next for Nodal

  • More scales - Pentatonic, Harmonic Minor, Whole Tone, Custom scale builder
  • Gesture vocabulary - Recognize conducting patterns (crescendo, staccato, legato)
  • Collaboration mode - Multiple performers controlling different lanes simultaneously
  • ML-powered assistance - Predict and suggest next notes based on musical context
  • Mobile support - Touch-optimized interface for tablets
  • 3D visualization mode - Depth dimension for velocity or expression control

Try it yourself

Experience the future of gestural music creation - conduct with your hands and watch music come alive!

Setup:

  1. Connect a virtual MIDI bus (IAC Driver on Mac, loopMIDI on Windows)
  2. Route to your DAW
  3. Enable hand tracking or drag the markers
  4. Make music!

Built With

Share this project:

Updates