MusicFromDaHeart

Inspiration

People spend hours curating playlists for every mood - one for focus, one for the gym, one for winding down at 2 a.m. But moods don't wait for you to build a playlist. Life moves faster than a Spotify queue, and the music that fit an hour ago rarely fits now.

So we asked what it would take to skip the curation entirely. Instead of picking a playlist that approximates how you feel, what if the music was generated from how you actually feel, in real time? That's what we built. A camera reads your vital signs through remote photoplethysmography - no wearable required - and your heart rate drives the tempo and intensity of a generative music bed. An Apple Pencil's pressure and motion shape the melody and timbre layered on top, so you can steer the sound as it forms: lean into the energy when you want a lift, or pull it down when you need to settle.

We used Presage's camera-based vital-sign sensing (with Polar BLE as a fallback) for the biometric input, Apple Pencil pressure and tilt data for the melodic layer, and fal.ai to generate the instrumental bed that everything plays over — all wired together through a local WebSocket audio engine.

What it does

MusicFromDaHeart generates music from your body in real time - no playlist required. A Polar Vantage reads your live heart rate and maps it to the tempo and intensity of the track. An Apple Pencil on iPad picks up pressure, position, and velocity, layering melody and timbre on top of that heartbeat-driven pulse. Together they steer the sound: press harder and the melody sharpens, calm down and the whole thing settles with you.

Rather than nudging a single track up and down, the app classifies your combined biometric and pencil state into distinct moods - calm, energetic, tense - and crossfades between genuinely different musical material, so a shift in how you feel produces a real shift in what you hear. You stay in control of the mapping:

Opposite-mood toggle inverts the whole system - when you want music to lift you out of a state instead of matching it
Dynamic/static mode freezes the music where it is, so you can hold a moment instead of following your pulse

Because the app is already watching your heart rate continuously, it also monitors for readings that fall outside a safe range - the kind of pattern that can signal cardiac distress. If it detects one, the music stops and an emergency alert is triggered.

How we built it

We split the system into three independently-owned modules: biometrics/ (heart rate), audio-engine/ (WebSocket server, playback, all mapping logic), and pencil-input/ (Apple Pencil capture), all connected by a single message contract that is frozen after Epic 0 (architecture stage), so builds never touched shared files.

Biometrics tested two real hardware paths side by side: phone-camera PPG (OpenCV peak detection) as the dependency-free fallback, and a Polar Vantage M relayed over BLE. Both sit behind one shared interface, so nothing downstream cares which is live. Pencil input runs as a Safari web app rather than native PencilKit, keeping pressure and velocity as the primary signal.

We used Warp's agentic terminal to implement each module from detailed prompts, and Claude to architect the system, scope each build step, and audit results before moving on. The instrumental bed and mood-zone stems were generated once via fal.ai and cached as committed audio files, so the live performance never depends on a generation call succeeding on stage.

Challenges we ran into

The original Vantage M (not the M2 or M3) has documented restrictions on third-party Bluetooth access to live heart rate. Polar themselves have said the original model's broadcast wasn't open to non-Polar apps. Rather than assume it would or wouldn't work, we built and tested a camera-based PPG (finger over the camera and flash) fallback in parallel from day one using Presage's vital-sign sensing, which pulls heart rate straight from a phone camera feed with no wearable involved. Both paths emit the same message shape into the audio engine, so swapping between them is a config change rather than a rewrite, and the demo was never dependent on an unverified piece of hardware.

The "jumpscare" problem. Early on we realized that simply smoothing and range-clamping the heart rate signal wasn't enough, a genuine sudden spike (or a sensor glitch) could still cause the music to lurch abruptly instead of ramping gracefully. We added explicit rate-of-change limiting on top of the existing smoothing to fix this.

Reliability under real, simultaneous input. Every module was tested in isolation first, but real problems only showed up once heart rate and Pencil input were both live at the same time, tempo and melody mapping fighting for the same parameters, timing assumptions that didn't hold. We built a dedicated integration pass specifically to catch and fix these interaction bugs rather than discovering them for the first time on stage. Making sure "crazy" never broke "working."

What we learned

Building for a live demo means designing for graceful failure, not just correct behavior, a system that's "usually right" isn't good enough when it's driving a live performance in front of judges.

Splitting a system by a single fixed contract, rather than by vague ownership, is what actually let the team build in parallel without merge conflicts or last-minute integration surprises.

Not every ambitious idea needs to make the cut, we scoped several "crazier" concepts (duet mode, live sign-language interpretation, distributed multi-device rendering) and deliberately cut or deferred the ones whose risk outweighed their value for a 36-hour build.