Pulse-Cam

Inspiration

Checking vitals almost always means touching a device to your body — a finger clip, a smartwatch, a chest strap. That falls apart in exactly the moments it matters most. A telehealth provider can't physically touch the patient on a video call. A caregiver is looking after someone who won't reliably wear a smartwatch. A newborn or a burn patient shouldn't have anything attached to their skin at all. We wanted to know if a camera everyone already owns could close that gap, and pull something as personal as a heartbeat out of the air, with nothing touching the person.

What we learned

The biggest lesson was that the hard part here was never really the code. It was the signal. Every stage taught us something new about the gap between "the algorithm is correct" and "the algorithm actually works." We learned that webcams don't hold a steady frame rate, especially once face tracking starts eating CPU, and that assuming a fixed 30fps produces a wrong answer silently. No crash, no warning, just a confident, incorrect number. We learned that naive green-channel rPPG is fragile enough that a lighting shift or a small head movement can completely swamp the real pulse signal. And we learned it firsthand, watching our own BPM read a few beats high, that a passing synthetic test doesn't guarantee a correct live result. Sometimes the bug isn't in your math. It's in which readings you choose to trust.

How we built it

We built the pipeline in stages, and verified each one live before moving to the next. Face detection and region-of-interest extraction came first, using MediaPipe FaceMesh to isolate the forehead and cheeks. Then signal extraction, averaging the RGB values in that region into a timestamped buffer. Then the DSP core: bandpass filtering, FFT, and peak detection. We kept that core pure and dependency-free on purpose, so we could unit-test it against synthetic signals before it ever touched a camera. Once the basic pipeline worked, we upgraded from a simple green-channel extraction to POS (Plane-Orthogonal-to-Skin), which combines all three color channels to cancel out motion and lighting artifacts that tend to move red, green, and blue together. We proved this mattered with an adversarial test: fed a fake motion artifact stronger than the pulse itself, the green-only method locked onto the artifact and reported the wrong rate, while POS recovered the true one. On top of that we added stabilization, median smoothing plus a physiological plausibility gate so a reading can't jump implausibly fast, and quality gates that refuse to show a confident number when the signal isn't trustworthy, freezing instead with a clear "hold still" message. What we ended up with is a guided 30-second pulse check with live diagnostic panels, so the system can show its evidence, not just its answer.

Challenges we faced

A couple stand out. The first was dependency friction on Apple Silicon. MediaPipe's API had changed between versions, so we had to rewrite our face-tracking layer against the current API in the middle of the build, with an OpenCV fallback ready in case that fix took too long. The second, and harder one, showed up after we'd built POS. Our live readings kept coming in a few BPM above manual pulse counts, even under good conditions, and our synthetic tests were all still passing. That forced some real detective work. We ruled out the DSP math itself first, sweeping synthetic tests across the target range and confirming the error was well under a tenth of a BPM. Then we used a record-and-replay setup to A/B test different regions of interest and extraction methods against the exact same footage, so we could isolate the actual source of the bias instead of just guessing at fixes. That process, proving what wasn't broken before finding what was, ended up being the most valuable part of the whole build. It's the same standard we try to hold every project to: don't claim a result you haven't actually verified.

Built With

• claude
• python