Resonance

Inspiration

The theme was "create art that wouldn't exist without technology," and most audio-reactive visualizers we've seen just scale a shape to volume — that's decoration, not something the tech makes possible. We wanted the sound itself to be the medium: not a track playing behind visuals, but the actual generative process. The idea crystallized around a simple question — what if the canvas never cleared? Every sound you've made becomes part of a visual record that literally could not exist without real-time signal processing.

What it does

Resonance listens to your microphone, an uploaded audio file, or a built-in procedural demo track, and turns the live frequency content into an evolving abstract painting. Under the hood, the Web Audio API extracts bass/mid/treble energy and overall amplitude every frame, which drive a noise-based particle flow field: bass warps the field's turbulence, mids shift hue, treble spawns bright "spark" particles, and amplitude scales overall motion. On top of that, a spectral-flux onset detector picks out actual beats and hits — not just loud moments — and fires a synced expanding pulse ring and particle burst on every detected hit. Because the canvas accumulates trails instead of resetting, freezing it produces a one-of-a-kind high-res still, and you can also export a full audio+video time-lapse of a performance.

How we built it

Vite + React + TypeScript + Tailwind for the shell, with the actual art engine as two framework-free modules: a SoundEngine (Web Audio wrapper — mic/file/demo input, FFT band extraction, beat detection) and a RenderEngine (Canvas 2D, requestAnimationFrame render loop, simplex-noise flow field, particle system). React only orchestrates state and UI — the audio and visuals never touch it in the hot path, which matters for keeping 60fps.

The rhythm layer uses spectral flux: each frame we half-wave-rectify the bin-to-bin increase across the full frequency spectrum, compare it to an adaptive threshold derived from a rolling history, and apply a refractory period so one hit doesn't double-trigger. Video export combines canvas.captureStream() with a MediaStreamAudioDestinationNode tapped off the same analyser, so the exported clip has real synced audio, recorded via MediaRecorder.

Everything is static and client-side — no backend — deployed to Vercel straight from GitHub.

Challenges we ran into

Getting beat detection to feel musical instead of twitchy took real tuning — raw spectral flux fires on every tiny fluctuation, so the adaptive threshold and refractory window needed several passes against real audio before hits felt intentional rather than noisy. We validated the detector against synthetic test signals with known beat timings to confirm it before trusting it against real music by ear.

Browser autoplay/gesture-gating was another one: AudioContext.resume() silently hangs without a genuine trusted user gesture, which shaped the whole permission-request flow. And keeping frame time stable on mobile required device-tier heuristics (particle budget and device-pixel-ratio scaling) since a naive implementation cratered on phones.

Accomplishments that we're proud of

Real spectral-flux onset detection — not amplitude-thresholding — running smoothly in the browser at 60fps alongside a live particle system. Also proud that video export captures genuinely synced audio and video of a live generative performance, not just a static frame, which turns the piece into something you can actually share as it happened rather than a single freeze-frame.

What we learned

How much perceptual quality in "reactive" art depends on separating true onsets from raw loudness — the difference between an art piece that feels like it's listening versus one that's just bouncing to volume is almost entirely in that detection layer, not the rendering. Also relearned that browser audio/media APIs are powerful but full of gesture-gating and autoplay edge cases that only show up once you test on a real phone on a different network.

What's next for Resonance

More curated palettes and the ability to save/share a session's parameters as a link; a MIDI/instrument input mode alongside mic/file; and possibly a collaborative mode where multiple mic sources shape the same shared canvas in real time.