Inspiration

You don't need expensive hardware and DAW software, all you need is your browser and your music (MIDI) + insprirations / creativities.

What it does

3D / 4D Visualization

After selecting your existing sound track (uploading a MIDI file), we visualize all the notes (pitch, starting time, duration, velocity) in the 3D world within your browser, using WebAudio and WebXR (Web Mixed Reality) technology.

Interactivity Then you can use your VR/XR glasses + controllers to interact with each note. In our roadmap, any parameters can be changed, transformation applied to suit your needs.

How we built it

• React Three Fiber (R3F) --> Three.js (WebGL framework)
• React Three XR -> WebXR, browser support for Virtual Reality (VR) and Augmented Reality (AR) devices

Challenges we ran into

• new frameworks, wrappers of wrappers
• functional programming to reference objects to interact with, precision timing, animation (UseRef, UseFrame, UseEffect)

Accomplishments that we're proud of

• First part is working!
• Our code is compatible with the latest Android XR announced by Google / Samsung 3 days ago!

Audio Support

• Basic sound out when observing a note
• MIDI OUT!

What we learned

• How to use React Three Fiber, and React Three XR
• How to use animation framework gsap
• How to reference objects
• How to detect collision (for 3D object interactivity)

What's next for DAW-XR: Mixed Reality Digital Audio Workstation in a Browser

• Interaction code complete
• Transformation functions
• Support both browser interaction +. XR glasses!

Built With

• react.js
• react.three.fiber
• react.three.xr
• three.js
• typescrpt