Inspiration
The idea for Musaic stemmed from our team’s shared passion for music and the passion to bring sounds into the physical world in an interactive and artistic way. We were intrigued by the idea of creating an engaging and wonder-inducing synesthesia experience that blends art, color and sounds, allowing users to not just hear music but see and shape it. Our diverse backgrounds – including one team member with classical music training – pushed us to explore how technology could make sound manipulation an immersive, visual, and tactile experience. Musaic is our team’s idea of a world where soundscapes are living works of art.
What it does
Musaic is a mixed reality, multiplayer experience that transforms the play environment into an interactive audio-visual canvas. Using 3D stylus controllers, multiple users can pick from an array of colorful brushes to collaboratively paint a sound sculpture - a Musaic - that can both look and sound beautiful. Each available brush is a different visual style, and also corresponds to a different music track of a song, such as the percussion track or the guitar track. The users are placed in the middle of a large circle. The edges of the circle represent a looping music timeline. You can see the time running continuously around the circle. By drawing colorful lines along that circular timeline, friends can collaboratively arrange a song by choosing how loud the different tracks play at different moments in the timeline. The resulting Musaic is both a feast for the eyes and for the ears - and it’s super fun to create together!
How we built it
We built Musaic for the Quest 3 and the Logitech MX Ink stylus controller. The app was built using Unity and the Meta Quest SDK (including the MR Utility Kit and the co-location technology for enabling the multiplayer experience). The app is experienced in passthrough mode, with users being able to easily see each other and their creation. The Musaic audio-visual sculpture is shared between all players using co-location.
We started with an extensive brainstorming and designing (low-fi paper prototyping) phase, in order to narrow down on the exact user experience. We identified a plan for an MVP prototype and identified stretch goals. Then, we worked in parallel on separate components of the project, namely: Research and deliverables preparation Co-location Drawing of lines with the stylus controller Shaders for displaying the app elements in a beautiful way Audio processing
Tools used for collaboration included Discord, GitHub, Google Drive, ShapesXR, and TLDraw. Four team members contributed Unity app code.
Challenges we ran into
Aiming for a multi-user experience, we needed to use co-location technology, which was heavily impacted throughout development by both network issues, and lighting / space tracking conditions (for the multiplayer to work, the headsets need to detect that they are in the same space, and it’s sometimes not obvious they will!).
The co-location also increases the complexity of the code in all aspects, because all key elements of the app logic need to be synced over the network.
Working on several different components in parallel has made it challenging at times to integrate the features into one working app - especially in the short timeframe! So some features we developed (such as cool shaders for the Musaic) didn’t make it into the final presented version.
Accomplishments that we're proud of
We’re really happy with the design we came up with, and how much of it we managed to implement in the short timeframe, resulting in a prototype that quite powerfully demonstrates the appeal of the app! The shared process of painting and at the same time generating music is a really fun experience - you really don’t want the painting, composing, and even dancing to end! The interaction has been well-received by our test users who praised how engaging it is!
What we learned
- how to work with music in Unity
- how to work with MX Ink for drawing
- how to implement co-located experiences on the Meta Quest 3
What's next for Musaic
- enable users to share their creations to social media with viral potential
- add more cool visual effects (shaders)
- build a marketplace for brushes so that creators can make audiovisual brushes that users can add to their inventory
- allow users to record their own samples from their environment and have additional control over the music if they want to (but still keep it friendly for non-musicians!)
Log in or sign up for Devpost to join the conversation.