DELPHI - Play the Stars
Inspiration
Musical instruments have evolved for thousands of years, from percussion and strings to keyboards and digital synthesizers. Yet most instruments still require us to interact with physical objects or flat interfaces.
We wanted to explore a different possibility: What if space itself could become a musical instrument?
Inspired by the constellation Delphinus and the mythology of a dolphin rescuing the musician Arion, we created DELPHI, a playful world where music, movement, and celestial storytelling come together.
What it does
DELPHI lets anyone compose music using natural gestures in three-dimensional space.
Reach out to grab a glowing star. Move it vertically to explore different pitches, or horizontally to discover different instruments. Connect stars to build musical phrases and watch those connections illuminate as the composition plays.
You're accompanied by Delphi, a celestial dolphin that swims through your musical universe and responds to your performance. It can echo your notes, add harmony or percussion, or leave space for you to play solo.
Each performance becomes something you can record, revisit, and share.
You're not playing an instrument in space. You're playing space itself.
How we built it
We developed a browser-based spatial music studio, using Three.js for the interactive 3D world and Web Audio for responsive musical playback.
The experience supports mouse interaction, webcam-based hand tracking, and WebXR controller input on compatible devices.
We incorporated SATIE musical sounds, with spatial star interactions mapped to different notes and instruments. A musical sequencing system converts connected stars into looping phrases, while Delphi's accompaniment system can echo, harmonize, or add percussion.
We also built a recording and sharing experience, including performance playback, a gallery, downloadable media, and shareable links.
In parallel, we created a Lens Studio asset and interaction package for Snap Spectacles, including a rigged dolphin, musical stars, constellation assets, and hand-interaction scripts.
Challenges we faced
One of our biggest design challenges was finding a musical interaction that felt intuitive without requiring users to learn traditional musical notation.
We explored different ways to map height, movement, instrument selection, and star connections to sound, while ensuring players could hear meaningful results immediately.
Synchronizing music with visual interactions was another challenge. A glowing constellation should not merely decorate the music; it should represent what the player is creating.
Building across browsers and XR devices also introduced practical limitations around hand tracking, performance, and device compatibility.
What we're proud of
We built a playable spatial composition environment where stars become musical controls, connected shapes become phrases, and a celestial companion becomes a creative collaborator.
We also extended the experience beyond the initial interaction: performances can be recorded, played back, downloaded, and shared with others.
Most importantly, DELPHI gives people a way to explore musical expression without needing to know how to play an existing instrument.
What we learned
We learned that creating a new instrument isn't just about producing new sounds. It's about designing a relationship between movement, space, timing, and musical expression.
The most important question became: Can someone discover how to make music simply by reaching out and experimenting?
That principle shaped our interactions, visual feedback, and musical design.
What's next
We want to extend DELPHI into a fully native, hands-first AR instrument, explore richer spatial audio and musical collaboration, and allow multiple performers to build constellations together.
Our longer-term vision is an instrument that doesn't imitate the piano, guitar, or drums, but introduces an entirely new way to compose and perform.
DELPHI - Play the Stars.
