Immersive DJ

Be a DJ in Mixed Reality in your Sitting Room (or kitchen, or anywhere!). No hardware required, just grab your Meta Quest. Scratch, EQ, perform. mix - with easy to use decks.

Comment 1

Inspiration

DeadMau5, Skrillex, Rezz, Lane 8, VRChat clubbing scene - get people creating amazing mixes using intuitive hand interactions in MR.

Available web technologies, including the Immersive Web SDK, shared open APIs such as SoundCloud and the advent of AI coding/driven development mean it's possibly to rapidly develop what would previously have been hard to achieve quickly.

Having spent a lot of time developing for the Quest family of devices with Unreal, it's time to try something different.

Answer the question: can a full featured web based XR app be developed rapidly using the Immersive Web SDK alongside the latest AI tools?

Add the following due to this tech stack:

  • Hand tracking interaction from the IWSDK
  • DJing audio engine functionality using the web audio standard
  • Rapid iteration in XR due to web tech stack ( no packaging, replaced with a simple page refresh )

What it does

  • Easy to use DJing workflow to create seamless mixes
  • Search and play tracks from the SoundCloud service and/or our own Suno generated library.
  • Visualise and interact with the track library in immersive mode using hand tracking
  • Beat sync using vari-speed
  • Quality waveform visualizers
  • DJ EQ controls

How we built it

  • Immersive Web SDK
  • Hand Tracking APIs
  • Passthrough mode
  • Gemini 3 preview to create the initial app and audio engine in 2D with React
  • Cursor to then integrate the 2D app logic as a backend for the immersive UX using IWSDK
  • Most of the UI is built programmatically rather than using the spatial editor
  • Vercel for server side deployment and services.

Early Access/Known issues

  • SoundCloud is often rate limited so you may see just our internal music library in your results. Rate limiting resets within 24 hours
  • Turn your right palm inwards to face you and use the long pinch gesture to reset the orientation of the UI

Challenges we ran into

  • AI limitations, during coding parts of the app, the models routinely resorted to incorrectly using Object3D instead of the Entity component system from the IWDK, even when rules are defined.
  • Trying to dynamically build Entity hierarchies didn't work so had to parent everything at creation time
  • A fair amount of custom work involved in iterating over knob and slider interactions as these aren't part of the IWSDK (what would be common controls in other frameworks). So a custom interaction system was made. It looks easy but this kind of interaction is hard to do well.
  • Performance issues/blocking of the UI during song loading and analysis. Fixed with async web workers.
  • Performance issues with the IWSDK raytracing in distance grabbable mode means this got removed. (1ms per hand per item).

Accomplishments that we're proud of

  • A fully featured app delivered in a short timescale (we started the day Gemini 3 was released)
  • It's a fully working app created at low cost with great hand driven interactions
  • The waveforms look cool, the mixes sound sick.
  • Creating a music backend service hosting Suno generated tracks, added at the last minute (as a fallback during SoundCloud outages/usage limits)

What we learned

  • You can't use AI for everything (yet)
  • Some fixes and updates needed to the IWDK
  • We wanted to make a PWA and deployed it but there's an unfixable bug with Virtual Keyboard access that makes the main use-case (Search) unusable - you can try it here if you want https://www.meta.com/s/3iGfq8t2c.
  • Iterating fast with a MacBook is a joy compared to 10 minute cooks on a PC.
  • BUT WebXR performance means the UI look, feel and performance is some way off what a native game engine app gives you - bring on WebGPU.

What's next for Immersive DJ

  • Better beat analysis, at the moment it's not as good as industry standard DJ software
  • Spatial anchors, this was implemented but not completed in time.
  • Promote and release in January
  • Scenes or props as IAP, or audio reactive visualisers.
  • More music services to play from (ideally Suno and Beatport)
  • Spatial mixing toys -> more unusual ways to control the mix, throw a virtual ball around the room, ball position controls an effect or EQ. Mish mash of DJing with fun interactions/physics interactions with the room as collisions.
  • Auto mix between rooms, as you walk between rooms in XR, the mix seamlessly changes track, so each room gets its own soundtrack without breaking the groove. Or soundtrack to your life, once AR glasses can run WebXR.
  • Multiplayer. Should be an easy add in WebXR

Built With

Share this project:

Updates