Inspiration
Every year about 12 million people have a stroke, and most survivors live with a weak or clumsy hand. Add hand fractures, tendon repairs and arthritis, and millions of people are doing hand rehab at home. That rehab is repetition: hundreds of fist-opens, finger pinches and wrist turns a day, for months. Clinicians hand out a printed sheet, and most people quietly stop. There's no feedback, no visible progress, and nothing to look forward to.
Our earlier work on inclusive, safety-first adaptive exercise kept pointing to the same gap: people don't need harder workouts, they need a reason to do the gentle ones every day. Quest hand tracking sees 25 joints per hand with no gloves or controllers, so it can measure every rep. We asked: what if each rep grew something beautiful, and the progress you can't feel became something you can see?
What it does
Tend is a calm garden on a lap-height planter. Each 3- or 6-minute session has five short exercises that hand therapists commonly prescribe:
- 🌷 Bloom (fist ↔ open): the big flower opens and closes live with your hand.
- ✨ Seed Pinch (thumb to each fingertip): the target fingertip glows on your real hand.
- 🌿 Fan the Ferns (finger spread).
- 🌻 Turn to the Sun (forearm rotation): the flower turns to face the sun as your palm turns up.
- 🪲 Fireflies (seated reach and coordination).
Every rep plants a flower. The garden persists: it fills the planter, then spills into the meadow around you. A month in, you're sitting in a field you grew with your own hands. A range-of-motion chart, a gentle streak and an adaptive coach give you reasons to come back.
Built for hands that don't work perfectly
- Your range, not a "normal" range. A 30-second calibration measures your comfortable open hand, spread and fist. Every goal is a percentage of your range.
- Never fail. If you're clearly trying but can't reach a goal today, Tend lowers it in the moment and tells you kindly. Between sessions, the coach raises or lowers each exercise's difficulty.
- Mirror therapy without a mirror box. If one hand barely moves, Tend reflects the healthy hand across your body's midline and draws it as a golden hand in place of the affected one.
- Dwell-to-press. Every button can be poked or activated by just holding a fingertip near it. Big targets, high contrast, spoken and captioned coaching.
- Pauses itself when your hands leave view or the headset comes off, and resumes when you're back.
- Private: no account, and nothing leaves the headset.
Keepsakes from your real life
We connected Tend to our Camera to 3D app, which turns a phone photo into a textured 3D model on your own computer. Photograph something you love, such as a grandchild's toy or a favourite mug, and it appears on your planter. Pinch it to pick it up and turn it in your hand. It's motivation and fine-motor practice in one.
How we built it
Tend is a WebXR app built with three.js and the WebXR Hand Input API. It runs instantly in the Meta Quest Browser in VR or passthrough MR, with no install.
- Gesture layer: turns 25 joints per hand into scale-normalised measures: per-finger curl from joint-angle sums, grip, abduction angle projected onto the palm plane, thumb-to-fingertip distances, and the palm normal.
- Rep engine: two-phase state machines with hysteresis on smoothed metrics, plus live "never-fail" goal adaptation.
- Fully procedural world, with no downloaded assets:
- a physically based sky with image-based lighting and a sunrise opening
- soft shadows
- 26,000 wind-animated grass blades in one instanced draw call
- sculpted, instanced flowers
- a reflective pond with butterflies
- fresnel "spirit hands"
- frosted-glass UI panels
- Audio: synthesised with WebAudio (pentatonic chimes that climb with each rep).
- Desktop hand simulator: produces the same 25-joint data. Paired with an autopilot that plays a full session, it let us test every flow without a headset and render our demo video frame by frame.
Challenges we ran into
- Measuring impaired movement fairly. Fixed thresholds fail people with limited range. We moved everything to per-user calibration and adaptive goals.
- Tracking noise vs. real reps. Smoothing plus two-phase hysteresis stops double counting without making the app feel laggy.
- Mirror therapy geometry: reflecting a hand across a plane anchored to the user's head and facing direction, so it lines up in place of the affected hand.
- Beauty on a mobile GPU: instancing, a tight shadow frustum around the planter, lower grass density in the headset, and updating only plants that are growing.
What we learned
Accessibility isn't a feature you add at the end. Designing for the weakest hand made Tend calmer and clearer for everyone. We also learned how much emotion a simple visual metaphor carries: a flower opening with your hand says "you did this" better than any number.
What's next
- Clinician mode: therapists set the prescription and receive progress reports.
- More exercises: tendon glides, finger tapping, wrist flexion.
- Co-op visits: family members join and water the garden with you.
- Seasons, so long-term users see time pass.
- A pilot with a stroke-rehabilitation clinic.
Tend is a wellness companion, not a medical device. Move only within a pain-free range and follow your clinician's advice.
