AI-DJ

AI-DJ reads the room's motion & sound from phones no recordings, just live vibe data and mixes music in real time. No playlist, no DJ booth: the crowd's energy shapes the set itself.

Comment

Inspiration

Most "AI DJ" projects we saw were really just smart playlists pick the next track, crossfade, done. We wanted the room itself to be the input, not just a listener. The question that stuck with us: what if the crowd's own movement could steer the music, live, instead of anyone picking songs at all?

What it does

Audience phones join over a browser (no app install) and read two things locally: how much the room is moving (camera) and how loud/rhythmic it is (mic). No video or audio is ever recorded or sent anywhere each phone reduces its sensor data to a handful of numbers a few times a second and discards the rest. Those numbers get combined across everyone in the room into a single "room state," which drives a live mixing engine that builds the set in real time tempo, layers, and filter brightness all shift with the crowd. There's no playlist and no one at a booth. The room's energy is the DJ.

How we built it

Phones do lightweight on-device signal extraction (frame-differencing for motion, RMS/onset detection for sound) and stream small JSON vectors over WebSocket never raw media. A server aggregates across however many phones are connected into one room-energy signal, then a pure mapping function turns that into music parameters. The sound itself comes from a rule-based procedural engine (Tone.js) a fixed set of loops all locked to one musical scale, so nothing the crowd does can ever produce a wrong note. Parameter changes are quantized to the beat/bar grid instead of applying instantly, which is what keeps it feeling like a DJ reacting instead of a system glitching. We split the build two ways: one of us on the phone-sensor and audio-engine side, the other on the aggregation/mapping logic, each testing against fake data before wiring the real pipeline together.

Challenges we ran into

Mobile browsers throttle camera access the moment a tab gets backgrounded or the screen locks that turned out to be a bigger risk than any of the audio or signal-processing work. Camera/mic access also refuses to run at all without HTTPS, which isn't obvious until you try it on a phone over local wifi. And getting the music to feel responsive without sounding broken took real tuning snapping to a new tempo mid-bar just sounds like a bug, so we built in deliberate delay and smoothing rather than reacting instantly.

Accomplishments that we're proud of

Getting the full loop working live someone moves, the room hears it change, within a couple of seconds in one week with two people. And the fact that it's demo-safe: because the music is built from one fixed scale and a quantized grid, there's no live input that can make it sound actually wrong, only a little sparser or busier.

What we learned

A lot about where to draw the line between "on the phone" and "on the server" when bandwidth and latency both matter, how to make a rule-based system feel intentional rather than random, and just how many quirks mobile browsers have around camera/mic access that never show up until you test on a real device in a real room.

What's next for AI_DJ

Testing with a real crowd at scale, not just a few phones in a room. Adding a language/region-aware track layer so the vibe fits the crowd culturally, not just energetically. And exploring a light melodic layer on top of the current rhythm-and-texture engine for sets that run long enough to want more musical variety.

Built With

Share this project:

Updates