AuraBeat

Inspiration

AuraBeat started with a simple question:

What if missing a concert did not mean missing the experience?

People can miss live music for many reasons: weather, distance, cost, sold-out venues, or accessibility barriers. For Deaf and hard-of-hearing fans, a concert can also depend too heavily on sound.

We wanted to build something that makes music more than something you hear.

AuraBeat turns a concert into something you can see, feel, and interact with.

What it does

AuraBeat has two main experiences.

At-Home Concert

The user's screen becomes a virtual stage with synchronized lyrics, lighting, crowd effects, and a virtual performer.

The phone acts like a digital lightstick, while an Apple Watch adds a physical layer through haptic feedback.

One of our main interactions is:

GET READY → watch vibrates → WAVE → wrist movement detected → stage reacts

Instead of only watching a concert, the user becomes part of it.

Live Concert

AuraBeat can also react to real music playing around the user.

Using microphone-based audio analysis, we detect musical moments and translate them into Apple Watch haptics such as:

Beat
Bass
Build
Chorus
Drop

Lighting and haptics are processed separately, so the visual experience and physical experience can react independently.

This creates another way to experience rhythm beyond sound alone.

How we built it

AuraBeat combines a web concert experience, wearable hardware, real-time audio analysis, and AI.

The main experience is built with React, TypeScript, and Vite. The Apple Watch companion is built natively with Swift and watchOS, using motion sensors and haptic feedback.

For the At-Home experience, we created a synchronized music timeline that controls:

lyrics
stage lighting
virtual performer behavior
crowd effects
haptic cues
interactive WAVE moments

When the Apple Watch detects an intentional wrist wave, it sends a movement_detected event back to AuraBeat, allowing the virtual stage to react immediately.

For Live Concert mode, we built a Python-based audio analysis pipeline that listens to real microphone input, detects musical onsets, and translates them into real-time Apple Watch haptics.

AI and Data Integrations

We also integrated several AI and data technologies into AuraBeat:

ElevenLabs powers Nova, AuraBeat's conversational virtual performer and concert host. Users can speak naturally to Nova and personalize how the concert feels through voice.
Google Gemini API supports AuraBeat's AI layer and helps create more intelligent, contextual concert interactions.
Tiger Data supports the data layer of the project, helping us manage information used by the concert experience.

Together, these technologies help AuraBeat move beyond a traditional media player into an interactive concert experience that can listen, respond, and adapt to the audience.

We designed each device to have a different role:

Screen = concert
Watch = feel + control
Body = interaction

Challenges we faced

The Apple Watch integration was our biggest challenge.

Our original WebSocket connection worked in simulation but did not work reliably on the physical Apple Watch. Instead of abandoning the hardware integration, we redesigned the connection using HTTP long-polling between the watch and our Vite relay.

We also discovered several hardware-specific problems that did not appear in simulation.

For example, an epoch timestamp stored as a normal integer caused a crash on the physical watch architecture, so we changed it to Int64.

Motion detection was another challenge. Our first wave threshold was too strict for real wrist movement. We collected real motion data, tested different thresholds, and tuned the detector until a physical wrist wave could reliably trigger the concert without creating too many false positives.

Finally, we had to make sure our Live Concert haptics did not interfere with the At-Home interaction system. We separated the lighting, haptic, and movement paths so each mode could work independently.

What we learned

The biggest thing we learned is that building an accessible experience is not just about adding an "accessibility mode."

Accessibility can be part of the main experience from the beginning.

Haptics, lyrics, light, motion, and visual rhythm are useful for Deaf and hard-of-hearing users, but they can also make concerts more immersive for everyone.

We also learned how different real hardware can be from a simulator. Many of our most important fixes only became visible when we tested AuraBeat on a physical Apple Watch.

Most importantly, we learned that interaction can change the feeling of digital entertainment. Instead of pressing Play and watching passively, AuraBeat lets the audience affect what is happening on stage.

What's next

We want to continue building AuraBeat into a more personalized concert platform.

Future ideas include AI-powered performer interaction, more advanced music analysis, multilingual experiences, richer haptic patterns, artist-created virtual shows, and shared multiplayer concerts.

Our goal is not to replace live concerts.

It is to make the feeling of being part of one available to more people.

AuraBeat — A concert that feels you back.