Rehabverse

Inspiration

Most people leave physical therapy with a printed sheet of home exercises, and most of those sheets end up in a drawer. The exercises are repetitive, there's no feedback, and it's hard to tell if you're doing them right. We wanted to keep the plan exactly as the PT wrote it but make doing it feel less like a chore.

What it does

You upload your PT's home exercise sheet (PDF or photo). Gemini reads it, and you check every field before anything is used. Each exercise gets its own small 3D world: squats draw water from a well to grow a garden, arm raises help birds take flight, balance holds stack stones by a lake, and follow-along exercises send moons into orbit. Your webcam counts reps, you can say "Start" to begin hands-free, and Nova, a voice companion, keeps you company. After each session you log how it felt, and you can download your progress to share with your PT.

How we built it

Next.js, React and Tailwind for the app, Gemini to extract the plan, MediaPipe Pose for tracking in the browser, three.js for the worlds, the Web Speech API for voice start, and ElevenLabs for Nova's voice. Progress and schedules are stored locally on the device.

Challenges

Counting reps fairly was harder than expected. A fixed squat depth doesn't work for someone in rehab, so reps are measured against each person's own starting position. We also had to be careful never to change the prescription or give medical advice, and merging two codebases built in parallel overnight took real work.

What we learned

We learned how noisy pose tracking is in real rooms, how much small feedback matters for motivation, and how to design health tools that support a clinician's plan instead of replacing it.