Sparks

Bring music to life

Inspiration

I’m inspired by the way that immersive media allows people to experience a kind of awe inspiring magic. My first experience with this was in my interactive VR short, Hickory Horned Devil, where viewers go from a storybook room through a portal into a magical world with a talking flower. People audibly gasp with wonder.

Music has a similar ability to transport people, it’s a perfect vehicle for interactive and immersive art. Sparks the Mixed Reality experience is inspired by an interactive art piece I’m creating in TouchDesigner where viewers interact with music and projected visuals through a webcam. I love the way that mixed reality allows things to exist in this wonderful place between the real world and the fantasy world. This project built for the Meta Horizon Start competition seeks to enrich the music listening experience by bringing visual magic into your own home.

What it does

Sparks provides viewers the opportunity to interact with audio reactive visuals. Play with kinetic audio reactive toys, decorate your space with audio reactive lamps and matching sconces. Set an audio reactive potted plant on the table and watch it dance. Choose from any of the songs that come preloaded, or enable the microphone to listen to your own! (use an external mic for best results.) Explore a new way of enjoying your favourite music, the music that comes preloaded, or performing your own music!

How we built it

I built it in Unity using simple assets made in Blender. I used both audio analysis and beat detection so that there are multiple ways of visualizing the music. Audio reactive particle systems and translucent shaders make good use of passthrough mixed reality and the URP by letting the real world show through virtual object. Hand tracking lets players intuitively pick things up and move them around.

Challenges we ran into

Everything ended up harder than expected. In the past I have worked with teams with a variety of skillsets, but doing this solo was extra hard. I especially struggled with the UI system because that is something I don't have a lot of experience with. Getting the hand grab interactions workable was another big challenge. I wanted people to be able to intuitively grab things and I didn't want to limit their options, but I also didn't want the various colliders to bump into each other. Fine tuning that took a long time.

Accomplishments that we're proud of

I think it's genuinely fun! I had a great time playing with it. At one point I was placing the chandelier on the ceiling by a mirror and I found myself surprised that it didn't show up in the mirror!

What we learned

I learned so much. Each time I had someone test it I found a new problem and a new surprise. I had friends and family test things out and they ended up loving playing with the kinetic items much more than I expected.

What's next for Sparks

I'd like to add in more songs. I'd also like to add in an inventory system where players can enable and disable the objects. I'm also planning on adding colour variations: holiday themes, bright dance themes, more sedate themes, etc.

Once the hardware is out, I think Sparks would be great for smart glasses as well.