Revera's Magic Drum

Pick your music genre and tailor the experience to your preference!
Bang on any object and turn it into a musical instrument!
Be guided through the experience by Revera, the magic AI Fairy!
Record up to 4 tracks that you can playback on loop

Inspiration

Making music is fun but has a high barrier to entry. The equipment is costly, takes time to set up, and takes up a lot of room! As a team, we were all passionate about music but were aware of how many people couldn't purchase, maintain, or store an instrument. We were inspired to create a solution that makes music production more accessible and engaging for casual users.

What it does

Revera's Magic Drum uses AI to generate unique melodic sounds for every object in your room. Simply tap on an object to play music with your furniture. Like a real drum, the sound is dynamic, depending on how you hit it. You can record yourself and play it back in a loop. We currently support up to four layers.

How we built it

Revera's Magic Drum is built using Unity and Meta's Mixed Reality Utility Kit. The musical sounds are generated via Eleven Labs. We utilize the Room Setup data from the Meta Quest to identify the objects in the room and their descriptions. This data gets passed to Eleven Labs with a prompt to generate a musical tone representing the described object. The generated tone is played whenever we collide with the object via Hand Tracking. We use the collision data to determine the volume and pitch of the tone. These collisions are also tracked and recorded to support looping.

Challenges we ran into

We initially wanted Revera's Magic Drum to work with any object without room setup. However, due to the constraints of Meta's SDK, we currently only support static environmental objects defined before launching the app. While we always knew that music would be at the app's core, we pivoted our target audience a few times as we tried to define what this product would be used for. Inevitably, we kept it simple and targeted musician enthusiasts.

Accomplishments that we're proud of

We have so much to be proud of. As a team, we stayed productive and collaborative throughout the hack. We are proud that we produced a working prototype early on Friday, allowing us to test, iterate, and refine our idea continuously. We are proud to have worked with the youngest American Reality Hack participant (16) as one of our primary programmers!

What we learned

We learned a lot about working with AI and integrating it into MR applications to create unique experiences personalized to people and their environments.

What's next for Revera's Magic Dum

Revera's Magic Drum has reached v1.0 and will be available to the public for free! We hope that this app will inspire many more like it. Music generation apps like these have the potential to improve until they reach feature parity with industry-standard tools such as Ableton or Frootyloops.