RecoveryIQ — About the Project
Inspiration
I've watched practitioners do the same thing at every assessment — hold a goniometer to a joint, eyeball an angle, write it down on a clipboard, and hope the patient remembers to do their exercises at home. The Hydrawav3 device already captures biometrics. The webcam is already there. The AI models already exist. Nothing was connecting them into something a practitioner could actually use in a session. That gap felt solvable in a weekend.
What It Does
RecoveryIQ is a browser-based practitioner console that turns the Hydrawav3 wellness device into a full clinical intelligence platform. A patient walks in, selects their area of concern on a 3D interactive body map, answers five AI-generated voice questions, holds still for a 10-second webcam biometric scan, then performs a movement test in front of the camera. By the time they sit back down, the practitioner has a Claude-generated session plan, a kinetic chain body diagram, measured joint angles, and a protocol recommendation — all in under two minutes.
How I Built It
Frontend — React 18 + Vite + Tailwind CSS. The 3D body map uses React Three Fiber. Charts are Recharts. The live skeleton overlay is an SVG canvas absolutely positioned over the webcam feed, redrawn on every pose frame.
Computer vision — MediaPipe Pose (mp.solutions, Python 3.12) running in a FastAPI backend. Joint angles are computed with a visibility-gated formula: only landmarks with confidence $\geq 0.55$ contribute to the measurement. The shoulder flexion angle uses a three-point vector:
$$\theta_{\text{shoulder}} = \cos^{-1}!\left(\frac{\vec{u} \cdot \vec{v}}{|\vec{u}|\ |\vec{v}|}\right), \qquad \vec{u} = p_{\text{hip}} - p_{\text{shoulder}},\quad \vec{v} = p_{\text{elbow}} - p_{\text{shoulder}}$$
Range of motion per joint is accumulated across all visible frames:
$$\text{ROM}_j = \theta_j^{\max} - \theta_j^{\min}, \qquad \text{where } \theta_j^{(t)} \text{ is included only if } v_j^{(t)} \geq 0.55$$
This avoids polluting the measurement with occluded or low-confidence frames.
Biometrics — The rPPG pipeline extracts a forehead colour signal from the webcam stream, applies a bandpass filter centred on $[0.75,\ 3.0]\ \text{Hz}$ (45–180 BPM), and computes heart rate variability as:
$$\text{HRV}{\text{SDNN}} = \sqrt{\frac{1}{N} \sum{i=1}^{N} \left( RR_i - \overline{RR} \right)^2}$$
No contact hardware. No wearable. Just a standard webcam.
AI layer — Claude claude-sonnet-4-6 handles three separate jobs:
- Writing the practitioner session brief from intake signals
- Generating a structured kinetic chain analysis — pattern name, per-structure role, and chain description — returned as typed JSON
- Powering an in-console chatbot with the full assessment loaded as system context, supporting multi-turn conversation
Kinetic chain visualisation — Claude's JSON output is matched against a lookup table of 28 named muscle regions, each mapped to SVG ellipse coordinates in a $200 \times 420$ viewBox. Regions are colour-coded by dysfunction role:
| Role | Colour | Meaning |
|---|---|---|
| Primary | 🔴 #EF4444 |
Root restriction driving the pattern |
| Secondary | 🟠 #F97316 |
Region forced into compensation |
| Stabilizing | 🟢 #22C55E |
Structure overworking to maintain control |
| Chain Flow | 🔵 #3B82F6 |
Distal adaptation endpoint |
Posterior muscles (glutes, piriformis, hamstrings, thoracic) render with a dashed stroke to distinguish front from rear anatomy without needing a second view.
Persistence — ROM snapshots, biometrics, zones, and session metadata are stored in SQLite via FastAPI and mirrored to localStorage for the progress trend charts on the Patient Journey dashboard.
Challenges
The angle formula was wrong. The original shoulder flexion code measured the shoulder → elbow → wrist angle — which is elbow flexion. Catching it required cross-referencing MediaPipe's 33-landmark index map and re-deriving the correct triplet from anatomy: hip, shoulder, elbow. A one-line fix that took an hour to find.
MediaPipe versioning is a trap. mediapipe 0.10.33 silently removed the mp.solutions namespace. The backend imported cleanly and crashed at runtime with an AttributeError. Pinning to 0.10.14 and Python 3.12 (3.13+ is unsupported by MediaPipe) fixed it — but only after learning that the error message pointed nowhere near the actual cause.
Chrome blocks autoplay audio. The voice intake uses browser TTS to ask questions. Chrome requires a user gesture before any audio can play. Solving this meant restructuring the entire question flow so each utterance fires from a button click rather than a useEffect, then managing a ref to prevent React StrictMode from double-firing the same question in development.
Real-time scoring without a ground truth. The /api/analyze-movement endpoint returns a quality score but only after recording ends. During recording, the frontend polls /api/analyze-pose every 4th frame, accumulates angle readings into a sparse per-joint array — missing frames are null, not 0 — then computes ROM and a quality score client-side as a live fallback:
$$\text{score} = 50 + \Delta_{\text{knee}} + \Delta_{\text{hip}} + \Delta_{\text{shoulder}} - \text{asymmetry penalty}, \qquad \text{score} \in [30,\ 95]$$
So the number on screen is always grounded in real camera data, never hardcoded.
What I Learned
Connecting computer vision, AI, IoT, and a real-time UI in one coherent product surface is mostly a problem of interface contracts — every layer needs to agree on what "no data" looks like. A sparse angle dict is better than a zeroed-out one. A null from the pose API means not visible, not straight. Getting those distinctions right is what separates a demo that lies from one that actually measures.
Log in or sign up for Devpost to join the conversation.