Ensemble

Whole setup
Scanning qr code

Inspiration

Wouldn’t it be fire if you could turn your audience into an orchestra?

At most performances, the audience watches while the sound comes from a stage or speaker system. We wanted to completely reverse that relationship. What if every person in the room became part of the performance?

We were inspired by orchestral conductors, DJs, interactive installations, rhythm games, and the magical feeling of waving a wand and immediately changing the world around you. Our goal was to make conducting music feel physical, intuitive, and accessible.

What it does

The conductor controls the orchestra with both hands. The webcam is the left hand: hold an open palm to play, hold a fist to pause, and pinch-drag to ride the whole room's volume and tempo like a fader. The physical wand is the right hand, and it works on positions, like a real conductor's frame: raise it high and harmony blooms; point it at the floor and the room hushes until you lift it; swing left for melodic runs; swing right and the chords break into arpeggios. Each pose is answered at the next bar line by an AI-composed part. From the console you can tap a phone to conduct only that instrument, or double-tap to solo it while everyone else falls silent.

Every change is temporary and theory-grounded, the way a conductor's cue is: the song breathes back to the score when your hands return to neutral. The result is not just a song played from several phones, it's a score being actively re-orchestrated across the room, bar by bar.

How we built it

Ensemble is powered by a Python asyncio server that coordinates the laptop, audience phones, computer-vision controller, and physical wand through WebSockets. The browser interfaces are built with HTML, CSS, JavaScript, and the Web Audio API. art was synchronizing audio across independent phones. The server never tells a phone to “play this note now.” Instead, it schedules notes roughly 150–600 milliseconds into the future using a shared server timestamp. Each phone repeatedly estimates the difference between its local clock and the server clock, then converts the scheduled timestamp into its own AudioContext timeline. This allows every device to prepare its audio locally rather than depending on a message arriving at the exact moment a note must play. engine, we use Mido to parse uploaded MIDI files into individual tracks, instruments, bars, melody lines, chords, tempo, and key information. The arrangement is then distributed between the connected phones, including synthesized percussion and different timbres for each section. vision interface uses MediaPipe Hand Landmarker in the browser. It identifies hand landmarks and recognizes poses such as an open palm, closed fist, finger counts, pointing, and pinching. We added gesture debouncing and hold durations so that briefly moving through a pose does not accidentally pause the entire performance. wand is built with an Arduino UNO Q, a Modulino Movement IMU, and a Modulino Distance time-of-flight sensor. The UNO Q contains both a real-time microcontroller and a small Linux computer. The microcontroller continuously reads the sensors and controls physical feedback, while the Linux side manages Wi-Fi communication with the laptop. Keeping musical interpretation on the laptop allowed the wand firmware to remain simple and reliable.

The musical intelligence is two LLMs we fine-tuned this weekend on Freesolo. A decision model maps gesture features to musical intent, and a bar-composer (a Qwen3.5-4B LoRA) receives the key, current chord, bar melody, and a style directive and generates one bar of notes as JSON, live, every bar. The training pipeline is the part we're proudest of: we wrote deterministic music-theory generators (voice-led chord pads, passing tones, Alberti arpeggios, approach runs), and every training example was validated against the exact programmatic reward we use for reinforcement learning — format, grid, key, and style adherence — with anything scoring under 0.95 rejected at build time. The resulting model scores 0.99 on held-out prompts. For latency, we went further: we merged the LoRA weights directly into the base model (verified bit-exact, tensor by tensor) and deployed it on a dedicated Fireworks H200 running FP8, measuring 453 ms per composed bar — ten times faster than shared-pool serving, fast enough to answer a gesture at the very next bar line. Everything the models emit is sanitized by the theory engine, and because the models were trained on that engine's output, it doubles as a bit-identical fallback: if every network on earth fails mid-show, the music doesn't. We use Gemini 2.5 Flash as an arranger; raw YouTube transcriptions are phantom-note soup, so Gemini rewrites them into clean melody/bass/harmony parts in our grid format.

The largest challenge was synchronization. Even if every phone receives the same WebSocket message, they will not necessarily produce sound simultaneously. Network delay changes constantly, phone clocks drift, and different devices have different audio-output latency. We had to separate message-arrival time from musical playback time and build a clock-synchronization and look-ahead scheduling system instead.

Browser security was another unexpected obstacle. Mobile browsers only expose motion sensors and certain camera functionality through secure contexts. That meant supporting separate HTTP and HTTPS/WSS paths, handling local certificates, and ensuring ordinary audience members could still join without dealing with certificate warnings.

Hardware integration was also difficult because the wand, server, music engine, and interface all had to agree on the same protocol. We designed the wand as a stream of raw sensor data rather than hard-coding musical commands into its firmware. This made debugging harder initially, but ultimately gave us much more flexibility.

Finally, translating movement into music was not as simple as mapping “left” to one effect and “right” to another. A gesture has speed, size, direction, height, duration, and context. We had to make the controls expressive without making them unpredictable—and powerful without requiring the conductor to memorize dozens of commands. The hardest fight was the IMU itself. Our gyro reads ~1°/s of bias at rest and the sensor is mounted rotated inside the wand, so "up" lives on an axis no one expects. Clever motion detection failed for hours until we stopped theorizing and recorded the raw sensor stream while performing a scripted pose sequence: the data revealed the bias, the mounting, and one genuine math bug (per-axis gyro integration breaks when the wand tilts 90°, scrambling the heading).

Accomplishments that we’re proud of

We are especially proud that Ensemble combines several difficult systems into one coherent experience: distributed networking, synchronized browser audio, MIDI analysis, computer vision, embedded hardware, real-time interaction, and AI-assisted music generation.

We built the system so the same server can accept multiple interchangeable conductor inputs. The physical wand, a phone’s motion sensors, and webcam-based tracking can all feed into a shared trajectory and gesture pipeline. This allowed us to test the music system before every piece of hardware was available and later connect the real wand without redesigning the server. roud of our failure-resistant architecture. Generated music is prefetched before the next bar, and deterministic candidates remain available whenever a model is unavailable. Phones can disconnect and reconnect, the last song and section assignments can persist, and the system includes automated tests for its handshake, clock synchronization, MIDI loading, gesture handling, and scheduling pipeline. tly, we created something that feels magical. You point at a phone, move the wand, and hear that part of the room respond.

We are proud of our failure-resistant architecture. Composed bars are prefetched before they're needed; the deterministic theory layer plays identical-by-construction parts whenever a model request is slow or fails; phones can disconnect and reconnect; songs and calibration survive restarts; and the pipeline is covered by automated tests, from clock sync and MIDI loading to gesture classification and the reward function itself. We're equally proud of the training discipline: every music-generation capability was painstakingly validated by ear as well as by judge devices to ensure quality.

What we learned

We learned that distributed audio is primarily a timing problem, not simply a networking problem. Sending packets quickly is not enough. Every device needs a shared understanding of when an event should happen and enough advance notice to schedule it locally.

We also learned that AI works best here when it collaborates with deterministic systems rather than replacing them. The model can provide creativity, contour, and musical variation, while the traditional engine guarantees timing, key, register, and playability.

Building the wand taught us to keep hardware responsibilities focused. The wand should sense motion, communicate reliably, and provide feedback. The laptop is better suited to interpreting gestures and making musical decisions because its logic can be updated without reflashing the hardware.

Finally, we learned that intuitive interfaces still require constraints. Conducting should feel free, but the system needs gesture segmentation, debouncing, musical quantization, and predictable fallbacks to transform that freedom into something that sounds intentional.

What’s next for Ensemble

Our next priority is improving synchronization over longer performances by estimating both clock offset and clock drift. We also want to add automatic acoustic calibration so Ensemble can compensate for the different speaker latency of each phone without requiring manual adjustment.

We plan to record more conducting sessions and use them to personalize the gesture-recognition and musical-decision models for individual conductors. This would allow Ensemble to learn how a particular person expresses ideas such as tension, calmness, energy, or anticipation.

We would also like to support larger audiences, richer instrument libraries, more advanced MIDI editing, collaborative conducting, and venue-scale installations. Eventually, an artist could distribute instruments across hundreds of phones and compose a performance around the physical layout of an entire room.

Our bigger vision is to transform concerts from something an audience watches into something the audience physically becomes.