Neural Synesthesia

Inspiration

Honestly, I wanted to combine sound and visuals into a single gesture. Traditional digital instruments require physical hardware, but I wanted to remove all barriers between the creator and the art. Built for Hack the Arts, Neural Synesthesia turns mid-air hand gestures into live music and generative visuals right in the browser—no extra hardware needed. The visual language is inspired by early 20th-century avant-garde movements — Suprematism, Constructivism, and the Bauhaus school — where geometric primitives and bold color fields become the vocabulary of expression. Every movement you make paints sound onto a dark canvas.

What it does

Neural Synesthesia is a browser-based Single Page Application that bridges the gap between physical movement and digital sound. Using your webcam and the MediaPipe Hands AI model, it tracks 21 key points on each hand and translates your gestures into musical notes, filter modulation, and abstract visual compositions — all in real time, with zero hardware beyond a computer and a camera.

How we built it

Built with React 19, TypeScript, and Vite. We integrated MediaPipe Hands for real-time webcam gesture tracking, Tone.js for polyphonic synthesis, custom effects, and audio recording, and a lightweight Canvas 2D API loop running at 60 FPS for dynamic, Bauhaus-inspired visual feedback. State between vision, audio, and graphics is synchronized via Zustand.

Challenges we ran into

I encountered significant performance bottlenecks while simultaneously running MediaPipe hand tracking, complex Tone.js audio synthesis, and real-time Canvas 2D rendering at 60 FPS. I solved this by decoupling the computer vision processing loop from the main rendering loop using optimized Zustand state snapshots. Additionally, overcoming Web Audio API autoplay restrictions and camera coordinate jitter required implementing an intuitive start splash screen and smooth coordinate interpolation for pitch stability.

Accomplishments that we're proud of

I am proud of creating a fully accessible and highly expressive audiovisual instrument that runs directly in any modern browser without requiring external MIDI hardware or special software. Furthermore, I successfully broke away from generic design clichés by delivering a distinctive, high-performance aesthetic inspired by Bauhaus and Suprematism that responds to physical gestures with near-zero audio-visual latency.

What we learned

Through this project, I mastered advanced Web Audio graph routing, polyphonic synth optimization, and custom effect management using Tone.js. I also gained valuable hands-on experience in optimizing MediaPipe landmark processing and designing decoupled architectures where vision, audio, and graphics communicate efficiently through Zustand.

What's next for Neural Synesthesia

Moving forward, I plan to implement WebMIDI API support so users can route their hand gestures as control signals to external digital audio workstations like Ableton Live or Logic Pro. I also aim to introduce multi-user collaborative jam sessions powered by WebRTC and allow users to switch between different artistic visual themes.

Built With

• 19
• audio
• git
• hands
• html5
• lucide
• mediadevices
• mediapipe
• react
• tone.js
• typescript
• vite
• web
• zustand