SynthSplat
An audio-reactive Gaussian-splat visualizer — real 3D-scanned worlds that breathe, ripple, and sparkle with your music, entirely in the browser.
Inspiration
Music visualizers haven't really evolved past abstract patterns. Gaussian splatting lets you capture real places as millions of 3D gaussians that render photorealistically in a browser. So we asked: what if the visualizer wasn't an abstraction of the music, but an actual place that moves with it? A candy shop pulsing on the kick, a jungle rippling with the melody.
What it does
Five splat worlds (Main, Candy, Jungle, Mushroom, Techno) you can fly through and crossfade between DJ-style. Live audio comes from any mic, interface, or system audio (Spotify via BlackHole), with Web MIDI knob mapping. Bass makes the world breathe, mids send concentric ripples, treble adds shimmer and sparkles — and a MusiCNN neural net classifies genre and mood every few seconds to drive the color palette, gated by live chroma (pitch-class) detection.
How we built it
Three loops on three threads:
Fast loop (~46 Hz): An AudioWorklet feeds frames to Meyda for spectrum, chroma, and centroid. Band energies pass through a gate → contrast curve → attack/release envelope so motion feels musical, not jittery. Beats are detected via positive spectral flux, firing when Φt>median+3.5⋅MAD\Phi_t > \text{median} + 3.5 \cdot \text{MAD} Φt>median+3.5⋅MAD over the last 43 frames — robust statistics survive dense mixes where mean/σ thresholds fall apart. Slow loop (~8 s): A Web Worker runs Essentia.js + TensorFlow.js MusiCNN on a rolling 10-second buffer, smoothing predictions with an EMA (si←0.65 si+0.35 s^is_i \leftarrow 0.65\,s_i + 0.35\,\hat{s}_i si←0.65si+0.35s^i) so one odd patch can't repaint the scene. Render loop: Three.js + Spark's dyno shader graph deforms every gaussian on the GPU. Bass dilates splats about the world centroid, c′=c0+r(1+0.18 bass A)\mathbf{c}' = \mathbf{c}_0 + \mathbf{r}(1 + 0.18\,\text{bass}\,A) c′=c0+r(1+0.18bassA); mids drive a traveling radial wave r^sin(3fd−3t)\hat{\mathbf{r}}\sin(3fd - 3t) r^sin(3fd−3t); treble applies spatially-hashed per-splat jitter.
Deployment is serverless: Cloudflare Pages serves the app, and a Pages Function streams the multi-hundred-MB splat files from a private R2 bucket with ETag and HTTP Range support — zero binaries in git.
Challenges
Keeping FFT analysis, ML inference, and rendering from blocking each other required strict thread separation with transferred buffers. Our first bass effect made whole worlds drift sideways — every deformation had to be reformulated as centroid-relative and net-zero. Raw FFT energy looked terrible until we tuned the full envelope chain by ear. And splat scans arrive in mixed coordinate systems, fixed with per-world quaternion flips.
What we learned
Gaussian splats are a shockingly good visualizer substrate — every "pixel" is a 3D primitive you can move and recolor individually. Perceptual tuning was the real work: most effort went into making motion feel like music, not into rendering. And the modern browser — AudioWorklets, Workers, WebGL, Web MIDI, WASM — is a legitimate platform for three concurrent real-time loops.
What's next
Tempo tracking to phase-lock ripples to the beat grid, WebXR so you can stand inside the worlds, and shareable presets bundling worlds + shader settings + MIDI maps. Connection to Marble API, more robust audio analysis for genre descriptions, and DJ avatar twin with emote capabilities.
Log in or sign up for Devpost to join the conversation.