HeartJam

User interface part 1!
User interface part 2!

Inspiration

The Gene-ius Hackers is a team of four Bioengineering students interested in how biology and hacking can combine. The idea for HeartJam was inspired from one member's experience in a research lab exploring generative music for non-communicative individuals. HeartJam transforms real-time heart rate data into an interactive musical experience so that listening to your heart can be reality!

What it does

HeartJam reads heart rate data from a smartwatch and uses the beats per minute to set the tempo. Users can select among a variety of instruments (keyboard, guitar, percussion, and bass) and select from different genre options (jazz, house, chill, rock). As heart rate rises or falls, the music speeds up or slows down to match the user's rhythm. To further integrate accessibility into our project, hand gestures signal the change of instruments, along with features like shuffling and music volume changes.

How we built it

To make HeartJam possible, we integrated:

Heart Rate API real-time heart rate data from FitBit sets the tempo for the music, adapting as the user's heart rate changes.
Mediapipe and OpenCV hand-tracking detects gestures for changing instruments, adjusting volume, shuffling the music genre, or play/stop all instruments.
Tone.js handles audio generation and manipulation.
React user-friendly frontend that integrates all components.

Challenges we ran into

Recognizing gestures to queue music proved to be more difficult than anticipated. The primary challenge that we faced was the integrating gesture detection with our frontend UI in React.

The other major challenge we faced was finding a smartwatch that allows real-time heart data extraction through a widely accessible API. We budgeted time for the integration of our sections, but the time required proved to be greater than anticipated!

Accomplishments that we're proud of

Integrating heart rate data onto our website
Syncing multiple instruments to the user's heart beat
Linking hand gestures to instrument controls
Creating our own logos and pixel icons for HeartJam

What we learned

For three of our four team members, this was the first hackathon that we participated in! There was lots of learning on front-end with React, along with OpenCV to process gestures on HeartJam.