-
-
Pinch and hold to start - no controllers. A ghost hand shows the exercise; you copy it while the app measures your form.
-
Fist open/close: live form score 84 with a 90% interval (±13), rep count, and a cue naming the finger that under-moves.
-
Finger tapping: live score 81 and a cue ("Open and close wider") when range of motion falls short.
-
Wrist rotation: form scored from range, tremor and rhythm across the 25 tracked joints per hand.
-
Session summary: per-exercise score, reps, range and tremor, with an overall score. Recorded in the Meta Quest 3 emulator.
Inspiration
Hand rehab exercises are boring, hard to do correctly alone, and progress is rarely measured. Quest hand tracking already reads 25 joints per hand, which is enough to judge form, not just count reps. And because it needs no controllers, it is usable by people who can't comfortably hold one - the people who need rehab most.
What it does
Put on the headset, pinch and hold to start - no controllers, no menus. A ghost hand demonstrates each exercise (fist open/close, finger tapping, wrist rotation); you copy it for 12 seconds while RehabLens scores your form live (range of motion, tremor, rhythm), counts reps, marks which finger under-moves, and tells you what to fix ("Bend your ring finger more"). Every score comes with a 90% confidence interval (about +/-13 points), so it says how sure it is. On a desktop browser with no headset, it runs a self-playing simulated-hand demo so anyone can try it from the link.
How we built it
IWSDK / WebXR in the Quest browser - nothing to install. Scoring runs on-device from the 25-joint stream (wrist-centred, hand-size-normalised). Models were trained and evaluated with subject-wise cross-validation on a synthetic kinematic exercise generator:
Classical signal features (range of motion, 4-12 Hz tremor, SPARC smoothness, jerk, rhythm): Ridge 4.6, XGBoost 3.3 MAE (points) Deep models: spatial Transformer with bone-tree attention + Mamba state-space temporal blocks, masked-joint and contrastive pretraining, cVAE augmentation, domain-adversarial head, one-class VAE Split-conformal intervals: 89.7% coverage at a 90% target (naive Gaussian: 28.6%) Integrated Gradients explanations: freezing the top-attributed joints moves the score 1.02 points vs 0.42 for random joints Few-shot low-rank personalization: 17.7 -> 9.8 MAE with 12 rated windows Distillation to a 112 KB int8 ONNX model, about 1 ms per 10 s window Challenges / honest findings
The deep models did NOT beat hand-crafted features on this data (14-20 MAE vs 3.3), so the shipped on-device scorer is the calibrated classical one (MAE about 6.4, 90% interval +/-12.8 on held-out synthetic subjects). We report the whole ablation ladder, including what failed. All numbers are on synthetic data and the demo video was recorded in a Meta Quest 3 emulator (IWER); the app has not yet been tested on a physical Quest and is not clinically validated.
What's next
Real recordings from patients with clinician ratings (record mode is already built in), domain adaptation to real hands, learned rep segmentation.
Built With
- github
- iwsdk
- javascript
- pages
- python
- pytorch
- scikit-learn
- three.js
- vite
- webxr
- xgboost