Inspiration

A friend of ours hurt their shoulder and was given a short list of home exercises. The doctor's advice was fine, but the routine was the same few moves every day, and it was easy to stop. Rehab only works if you keep doing it, and that is a motivation problem as much as a medical one.

Rhythm games solve exactly that problem: they make repetition feel like a challenge. We asked what would happen if every rep had to land on the beat, and a score told you how well you did it. RehaBeat is the result: an arcade rhythm game for home stretching and rehab movements, with a doctor's view on the other side.

What it does

You stand about two meters from your laptop. A target figure appears in the camera picture, and you match it on the beat: reach, hold, release.

  • Moves land on the music. Each move is placed on the song's beat grid: you reach on the count, hold for a few beats, and release. The song plays from start to finish whatever you do, so a missed move is a Miss and the next one comes round on the beat, as in any rhythm game.
  • Live feedback. A dot at the end of each limb the move scores turns from white to green once that limb is in place, and the target figure turns green when your whole pose matches. A short hint like "Left arm higher" points at the part to move. If part of you leaves the frame, the game asks you to step back.
  • Scoring. Each hold is Perfect, Good, or Miss. The results screen gives a grade, stars, combo, an estimated calorie count, and XP. Players who sign in with Google also get a place on the leaderboard.
  • The doctor loop. On /doctor, a clinician picks moves, rounds, per-joint limits, and a note. A patient links to their doctor with a join code. The patient's title screen shows that program, the session ends on its own, and the results go back to the doctor's page, which shows consistency, quality, and problem spots. If a patient pushes a joint past the doctor's limit, the game tells them to ease off. The doctor can also have Gemini draft a summary of the patient's sessions and a note, which the doctor edits before sending.

Pose tracking runs in the browser, and the camera video never leaves the laptop.

How we built it

  • Pose tracking: MediaPipe's pose landmarker runs in the browser. Each joint angle is read two ways, from the 3D world landmarks and flat in the camera picture, and judged by whichever is closer to the target, because the 3D depth estimate made straight limbs look 10 to 35 degrees bent. Angles are smoothed with a One Euro filter we wrote and unit-tested.
  • One clock: AudioContext.currentTime is the only clock. Audio is scheduled on it, and each camera frame is stamped and mapped onto it, so movement and music share one timeline.
  • Scoring: the frame score comes from the mean angle error against the move's tolerance:

$$ \text{score} = \operatorname{clamp}\left(100 - 40 \cdot \frac{\bar{e}}{\tau},\ 0,\ 100\right) $$

where ē is the mean absolute joint-angle error and τ is the move's tolerance. An error equal to τ scores exactly 60 (Good), and 85 or more is Perfect. A move's score is the time-weighted mean over the best three-quarters of its hold, so landing a beat late into a four-beat hold costs nothing.

  • Music: one command adds a song. Demucs splits it into drum and melody stems, librosa finds the beat grid, and the game places each move on that grid.
  • Calibration: four clicks measure the player's audio-to-movement offset, which scoring then applies.
  • Moves: three general exercises plus eight yoga poses derived from a pose-landmark dataset.
  • Doctor side: prescriptions are converted into ordinary moves (a joint limit turns a target into a lower one, or removes the move), and sessions record only numbers, never images. A demo clinic runs in localStorage with sample patients. The real clinic runs on Supabase, with row-level security rules we tested as a doctor, a second doctor, the patient, and a stranger.
  • Gemini summary: a Supabase Edge Function sends Gemini the patient's condition, program, scores, and problem spots (never their name) and returns a short summary, up to three changes to consider, and a draft note. The prompt is built on the server, and the key never reaches the browser.
  • Accounts: Google sign-in through Supabase Auth, with a leaderboard of signed-in players' best runs.
  • Front end: React, TypeScript and Tailwind, with Motion and GSAP for animation and synthesized arcade sound effects.

Challenges we ran into

  • Time constraints. We had 36 hours to come up with the idea from scratch and build the whole system: pose tracking, a rhythm engine, music, a game interface, and a doctor's page. We couldn't polish everything, so we had to decide early what mattered most. We chose the core loop, matching a pose on the beat and getting scored on it, and built the rest around it, even though it meant leaving some things unfinished, like testing the signed-in clinic across two devices.
  • Data. Rehab movements need target poses, and we had no dataset of our own to start from. We had to decide how to collect or source that data. We built the eight yoga poses from a public pose-landmark dataset, measuring joint angles from its photos and keeping only the joints that most photos agreed on. We wrote the three general exercises by hand. Because the targets come from small datasets and webcam estimates, not clinical measurements, we treat the angles as guidance. We also had no patient or clinician data, so the doctor's page runs on generated sample data in a demo clinic.
  • Scoring real bodies. Our first real players all scored a C, with more than half their moves a Miss, even when their form looked right. Running our pose model on photos of correctly held yoga poses showed why: the 3D depth estimate read straight elbows and knees as bent. We now read each joint both in 3D and flat in the picture, and score the best three-quarters of each hold.

Accomplishments that we're proud of

  • It feels like a game. Every reach and release lands on the song's beat, the edge of the picture lights up with it, and each finished hold bursts into Perfect, Good, or Miss with its own sound.
  • One shared clock for sound and movement. Camera frames are mapped onto the Web Audio clock, and a four-click calibration measures each player's timing offset. "On the beat" is measured to a few milliseconds, not guessed.
  • Scoring that is fair to real bodies. On 17 photos of correctly held yoga poses, our average score went from 67 (no Perfects, 4 Misses) to 83 (6 Perfect, 11 Good, no Misses) after the scoring fix.
  • Both sides of rehab. The patient plays a rhythm game, and the doctor gets a page to choose moves, rounds and joint limits. Sessions come back with consistency, quality and problem spots, and Gemini can draft a summary and a note for the doctor to edit. The game builds each session from the doctor's program, so the loop of prescribe, play, measure and adjust works end to end in our demo clinic.
  • Real, not mocked up. We built eleven moves (three general exercises and eight yoga poses derived from a pose dataset) and eight songs in three tempo groups, each split into stems with a beat map. We also wrote our own smoothing filter, with unit tests, a simulated player test and a browser smoke test that drives the whole flow with a fake camera.

What we learned

  • Audio and video need one shared clock, or "on the beat" has no meaning.
  • Game-feel details (dwell times, fades, sounds) matter as much as the algorithm.
  • Test the idea, not just the code. Our first design played the melody only while you held the pose. It sounded great on paper, but in practice players spent most of a run hearing only drums. Now the whole song plays and the score does the judging.
  • A doctor's view only matters if it is as simple as the patient's.
  • Be precise about claims. RehaBeat does not treat injuries or replace professional guidance.

What's next for RehaBeat

Test usability with patients and clinicians, improve the pose guidance, and have a clinician review the Gemini summaries. We also want to test the signed-in clinic end to end across two devices: its database and privacy rules are live, but so far we have only run the full loop in the demo clinic.

Built With

Share this project:

Updates

Submission history