Heartware

An intelligent wearable that transforms your body signals and surroundings into a real-time, interactive musical orchestra.

Comment

Inspiration

Have you ever noticed that when you walk with your headphones on, your steps naturally start matching the beat of whatever song is playing - and sometimes you even change songs to match your mood or pace. What if the music didn’t just influence your movement, but your movement created the music? Instead of picking a track first, your heartbeat, steps, and gestures could shape the sound in real time. With today’s AI music tools and sensors, it feels possible to turn everyday motion into personalized, living soundtrack.

What it does

Introducing Heartware - a sensor-powered and wearable adaptive music system that generates and controls music using your walking pace, gestures, heartbeat, and taps. Without needing to touch a screen, users can shape tempo, mood, and rhythm through natural movement. Google AI continuously transforms these signals into responsive, real-time music, creating a personalized soundtrack that evolves with your activity and environment.

How we built it

We leveraged the Lyra musical model built by Gemini API in order to integrate real-life environmental attributes and factors into musical tunes and outputs. On the hardware side, we wired together a photoresistor, heart-rate monitor, rotary encoder, inertial measurement unit, and LCD, powered by the Arduino UNO Q's dual MPU and MCU in order to bring everything together. The UNO would grab sensor values from the Arduino side into the Linux side through the bridge, and connect through the model running on a PC through sockets. These values would drive the music configuration and genre weights, the genres which the user can select between.

Challenges we ran into

While developing Heartware, an entirely new way of interacting with music, we faced the following challenges:

  1. Strap Adaptation – When we were using SolidWorks to CAD the design of the shell and the lid, we did not have the wrist strap. Therefore, we estimated the size of the slot for the strap to fit through. We adapted by cutting a velcro strap from Dollarama to fit through the slot.
  2. Component Mounting – We couldn’t find screws suitable for securing Arduino components inside the box. Instead, we used Blu Tack and tape to attach them to the interior walls of the shell. Faulty Sensors – Both the heart rate sensor and gyroscope were faulty, requiring troubleshooting and adjustments.
  3. Hole Placement in CAD – Determining exact locations for holes in the CAD design was difficult and time-consuming. Initially, we planned to use screws to fix each component into place. However, with a shortage of screws, we were not able to procure them for the assembly.
  4. Bluetooth Issues – Our initial idea was to have an isolated system able to run this generative model using the dual-systems on the UNO Q, but we encountered an issue at the final step when we were unable to connect to a Bluetooth speaker. As the ability to demonstrate changes in audio was integral to the hack, we had to pivot last minute to a different workflow involving a laptop on standby. Additionally, we were unable to use WIFI to transmit data due to the UNO's limited capabilities.

Accomplishments that we're proud of

We’re extremely proud that the Heartware prototype functions as a new interactive music interface and that we were able to successfully integrate the hardware components and enclosure despite multiple setbacks. However, we would like to improve the reliability of the sensors and make the internal mounting and casing design more precise and permanent in future iterations.

What we learned

This project strengthened our adaptability and creative thinking. We faced repeated obstacles during development, but each setback required us to adjust our approach and try new solutions. Working through these challenges taught us how to stay flexible, resourceful, and persistent when plans didn’t go as expected. It also significantly strengthened our technical skills in Arduino, CAD, and programming through hands-on design, building, and troubleshooting.

What's next for Heartware

Heartware is close to a prototypal version, needing only the sockets to fully implement the transmission of socket data in order to drive the AI, and a refined enclosure for the hardware. In the future, additional sensors could be implemented to get a better an even better gauge of the user's environment.

Share this project:

Updates