Synesthesia Deck

A mixed-reality VJ instrument where one performer turns music and physical faders into a shared spatial world.

Comment

Inspiration

Music has always had the power to dissolve the distance between people. In a club or performance space, a room moves to one rhythm while every person experiences a different inner landscape. Synesthesia Deck began with a question: what if those invisible feelings could spill into the room as a shared spatial world?

The project draws from synesthesia—sound becoming color, movement, and architecture—but it is not simply about making music visible. The aim is that fleeting moment when the boundaries between performer, audience, and image begin to soften. We no longer stand outside the image; we enter it together.

What it does

Synesthesia Deck is a mixed-reality VJ instrument for one performer and many viewers. A DJ or VJ uses a physical KORG nanoKONTROL2 while audience members wear Meta Quest headsets in the same room. Music and gestures reshape procedural scenes around the real space: code cathedrals, data cities, particle storms, tidal ribbons, mycelium colonies, deep-sea blooms, luminous gardens, ember flocks, origami mechanisms, moiré lattices, woven knots, resonant chimes, and prismatic tunnels.

Eight faders shape intensity, density, speed, scale, distortion, glow, glitch, and master effects. Eight scene controls change each world’s geometry, palette, and motion. One-shot and persistent effects include Orbit, Pulse, Twist, Mirror, Scatter, Strobe, Prism, and Freeze. A tactile DROP command contracts the world, flashes, releases a seeded explosion, and can move the whole audience into the next scene. A separate read-only spectator view automatically frames the show for people outside the headsets.

How we built it

We made the Mac the sole show authority because the live audio and physical controller were in the venue. A native Swift host uses ScreenCaptureKit to capture system output, Accelerate to analyze the audio in real time, and CoreMIDI to read the nanoKONTROL2. An authenticated Node.js and TypeScript service owns the monotonic clock, scene state, controls, seeds, and scheduled events.

Every headset estimates its offset from that clock and renders locally with Three.js, WebXR, and WebGL2. Only compact numeric audio features, controls, seeds, and timestamps cross the network—never raw audio, video, particle positions, or scene geometry. Deterministic procedural rendering lets independent browsers paint the same evolving world from their own physical viewpoints.

For mixed reality, we used a transparent Three.js rendering path with TSL bloom and luminance-derived alpha. Bright geometry and halos remain vivid while empty pixels reveal passthrough. The Vite frontend is deployed on Vercel over HTTPS, while the live Mac authority is reached over trusted WebSockets. Operator, headset, and spectator views are separate, so command credentials never need to appear in audience links.

We also built automated TypeScript and Swift tests for the protocol, clock behavior, scene determinism, reconnection, XR session handling, audio analysis, and MIDI mapping. The scene bank grew to 15 procedural worlds, all driven by the same musical features and tactile controls.

Challenges we ran into

The first major challenge was shared time. Two headset browsers have independent clocks, render loops, and Wi-Fi conditions, so simply sending “do this now” produces visible drift. We introduced an authoritative monotonic timeline, clock-offset estimation, future-scheduled events, deterministic seeds, and snapshot recovery after reconnects.

The second challenge was visual. Conventional bloom often paints an opaque background, which destroys the feeling of mixed reality. We had to preserve luminous halos while keeping the rest of the frame transparent and making the composition readable from moving, human-scale viewpoints.

The third challenge was operational: trusted HTTPS, WebSocket reconnects, macOS system-audio capture, display sleep, headset lifecycle, and fitting 15 scenes plus effects onto an eight-channel hardware surface. A live instrument has to recover gracefully, not just render a beautiful first frame.

Accomplishments that we're proud of

During the hackathon, Synesthesia Deck ran across four HMDs with a physical nanoKONTROL2 and live system audio. The audience shared the same scene, seed, timeline, and musical events. In one instrumented physical REC/DROP test on two of those headsets, the two measured clients applied the event within about 1.9 milliseconds of each other at the application-timing layer. We also verified recovery after temporarily disabling Wi-Fi on one headset: it rejoined the current state without replaying an expired DROP.

A 30-minute run exercised all 15 scenes, quality changes, and 20 DROP events while both headsets remained connected in MR. Medium quality sustained approximately 90 application frames per second at the median. More important than the numbers, the system became genuinely performable: moving a physical fader felt like sculpting a world around other people.

What we learned

Synchronized MR is less about broadcasting the same picture than composing a shared sense of time. Local rendering keeps the network light and lets every person see from the correct viewpoint, but it demands deterministic mathematics and a clear authority model.

We also learned how much a physical interface changes the artist’s relationship to software. Instead of editing parameters on a screen, the performer can listen, react, and shape the room by touch. That immediacy is central to the piece.

What's next for Synesthesia Deck

Next we want to harden venue operation on a dedicated local network, improve shared spatial alignment and moving-view composition, refine onset and tempo analysis for more musical material, and build reusable performance sets and cue authoring. We also want to explore shared spatial anchors, a persistent relay for easier deployment, and additional scenes designed together with musicians and audiences.

Created at Constellation Labs, Espronceda Institute of Art and Culture, Barcelona.

Share this project:

Updates