BodySense

Inspiration

Most health information lives inside the clinic. A sensor gets clipped on your finger, a number appears, and you leave without knowing what was measured or why it matters. Meanwhile, millions of people wear watches, rings and headbands that record their heart, sleep and brain activity every day, and get scores without explanations. People have more health data than ever, and less understanding of it.

We wanted to close that gap with something everyone already owns: a phone. Its camera, flash and microphone can pick up a heartbeat, a breath and the sounds of the heart. If we could show people their own body's signals in real time, and explain honestly what those signals mean and when to trust them, health education could happen anywhere: at home, in a classroom, or far from a clinic.

What it does

BodySense is a free web app that turns a phone into an interactive physiology lab, built around a 3D model of the human body.

Your own signals, live, with no extra hardware
- Fingertip: a finger over the rear camera and flash gives your pulse waveform, heart rate and heart rate variability (HRV).
- Face: a contactless scan reads your pulse from tiny colour changes in the skin, and the face visibly flushes with each amplified heartbeat.
- Microphone: pressed to the chest, it becomes a stethoscope that shows the "lub" and "dub" of your heart valves.
- Breathing and stress: breathing rate from the same camera signals, and a calm-or-stressed gauge from HRV, compared to your own baseline.
Every reading has a quality score. It rates beat shape, pulse clarity, rhythm, signal strength and stillness, and says in plain words what's wrong ("hold your hand still", "too dark"). If the signal isn't trustworthy, it doesn't show a number.
See where each signal comes from. Tap an organ on the 3D body and the camera flies there. The heart beats at your live heart rate and the lungs follow your breathing. A simulator lets students trigger a blink, a startle or a deep breath and watch every signal respond.
Connect devices you already own. Stream heart rate from a Garmin watch or chest strap over Bluetooth, or load Apple Health, Garmin, Oura and Muse exports to see trends.
AI explanations grounded in your numbers. Gemini turns a numeric summary (never video) into plain-language insights tied to the physiology, and never diagnoses.
Private by design. All signal processing runs on the device, and it installs like an app and works offline.

How we built it

BodySense is a static progressive web app in plain JavaScript, hosted on GitHub Pages, so anyone can open it on any phone.

Fingertip pulse (PPG). Each camera frame gives the mean red intensity. We band-pass it to $0.7$–$3.5$ Hz (42–210 bpm), find systolic peaks, and compute heart rate from the median inter-beat interval:

$$\text{HR} = \frac{60}{\operatorname{median}(\text{IBI})}\ \text{bpm}$$

HRV is the root mean square of successive differences between beat intervals, using beats from good-quality windows only:

$$\text{RMSSD} = \sqrt{\frac{1}{N-1}\sum_{i=1}^{N-1}\left(\text{RR}{i+1}-\text{RR}{i}\right)^2}$$

Quality score. Five components $c_k \in [0,1]$ (beat-shape correlation, spectral clarity, rhythm, perfusion, stillness) are combined with weights $w_k$:

$$Q = \frac{\sum_k w_k\,c_k}{\sum_k w_k}$$

A heart rate is shown only when $Q$ is fair or good and no single component is very low.

Face (remote PPG). A skin-colour tracker samples the forehead and cheeks. We use the POS method (Wang et al., 2017): normalize each colour channel by its recent mean, project onto two axes orthogonal to skin tone, and mix them so lighting changes cancel:

$$S_1 = G_n - B_n,\qquad S_2 = G_n + B_n - 2R_n,\qquad h = S_1 + \frac{\sigma(S_1)}{\sigma(S_2)}\,S_2$$

The pulse signal $h$ goes through the same processor with face-specific thresholds. Band-passed and amplified, it's also painted back onto the face as a red flush.

Heart sounds. Raw microphone audio (echo cancelling and noise suppression off) is filtered to $25$–$400$ Hz and turned into an energy envelope. Autocorrelation finds the cardiac cycle, and each sound is labelled S1 or S2 from its timing (systole is shorter than diastole) and loudness.

Breathing. From the light level under the fingertip pulse, or the tiny rise and fall of the head in face mode. Over a 32-second window we pick the strongest frequency between $0.1$ and $0.6$ Hz (6–36 breaths/min), and score quality as the share of in-band power at that peak.

3D and devices. BodyParts3D anatomy via the open-source Plethscape viewer (three.js), with a Muse headband fitted to the head by raycasting. Watches and chest straps connect through the Web Bluetooth Heart Rate profile.

Testing. Every signal has a simulator with scripted artifacts (motion, weak contact, glare, talking). We also generated a synthetic face video and fed it to the browser as its camera, to test the full pipeline end to end.

Challenges we ran into

The face signal is tiny. Each heartbeat changes skin colour by well under 1%. Early versions sometimes locked onto every second beat and reported half the real heart rate with confidence. We added safeguards and chose to show no number rather than a wrong one, and hid HRV in face mode because 30 fps video is too coarse for beat timing.
Phones fight you. Most can't run front and back cameras at once, iPhone browsers can't connect to Bluetooth watches, and camera auto-exposure drift looked just like breathing. We tested each shortcut and dropped the ones that gave confident wrong answers.
Honesty versus hype. Blood pressure and blood oxygen from a phone camera were tempting, but they need calibrated sensors or licensed, validated models. A health tool that overclaims does more harm than good, so we left them out and explain why.
Licensing. Some 3D head scans couldn't be redistributed, so we rebuilt the viewer to show open anatomy instead and credited every open dataset.

Accomplishments that we're proud of

A phone becomes a fingertip pulse sensor, a contactless face scanner, a stethoscope and a breathing monitor, with no app store and no extra hardware.
A quality score that explains itself and refuses to guess.
On simulated signals: fingertip heart rate within 1–2 bpm from 45 to 180 bpm, face within 1–3 bpm, heart sounds within about 1.5 bpm. A synthetic face video through the real camera pipeline read 66 bpm (true 66) and 15 breaths/min (true 15).
Every number connects back to an organ on a 3D body, so people see where a signal comes from, not just what it is.

What we learned

Signal quality matters more than the number. Knowing when not to trust a reading is the most important health-literacy lesson, and the hardest engineering problem.
The physiology is beautiful once you can see it: breathing speeds up your heart on every inhale, stress quietly steadies your heartbeat, and a blink is bigger than your brainwaves.
The browser is a surprisingly capable health-education platform, as long as you design around its limits.
Simulators let us move fast, but real bodies are messier. Validation is the next step, not an afterthought.

What's next for BodySense

Real-world validation against chest straps and clinical pulse oximeters across skin tones, ages and lighting, with results published openly.
Compare every source live: fingertip vs face vs watch vs microphone, with Gemini explaining which to trust right now and why.
Classroom mode for high-school biology and pre-med courses: the teacher drives the simulator, students follow on their own phones.
More signals: microphone breathing rate, cough and snoring detection, and guided breathing with live HRV feedback.
Accessibility: translations, screen-reader support and low-bandwidth modes for communities far from clinics.

BodySense is an educational tool, not a medical device.