Understudy: mirror exercise for a weaker hand

• 

  Understudy gives each finger its own assistance setting in a seated mirror exercise; this is the real app in Meta IWER.

• 

  The synthetic tracker model gives index range a 20.8° change margin; repeated measurements guide each finger’s assistance.

• 

  Hand Print measures three movements over three trials and shows an estimated error margin; the hands here are simulated.

• 

  Watch, Rehearse and Perform use 0%, sample-derived and 100% own-movement shares; these describe the rendered blend.

• 

  The coach explains each finger’s setting from the same measurements and margins that set assistance; history is simulated.

• 

  Next visit shows each finger’s next change threshold and 14 days of visits; this prepared sample keeps personal history separate.

• 

  Progress places an earlier pose beside the latest and compares change with estimated error; this is simulated history.

• 

  Desk Break runs spread, wave, dial and stones for two working hands at a desk; this capture uses simulated hands.

• 

  Pause freezes the routine while Resume and Stop remain within reach; this is the real paused app in Meta IWER.

• 

  178 unit tests, four input journeys and 60 retained room-placement checks cover software and emulation.

• 

  Meta hand tracking drives the mirror, measurements and controls; local records feed assistance and the next visit.

• 

  Meta IWER’s actual developer controls around the deployed app show the emulator used for these captures.

About the project

Inspiration

Understudy is a seated daily mirror exercise for someone practising with one weaker hand. Meta hand tracking drives the reflection, measures movement and operates the controls.

The moment I designed for is a morning at the same table: measure, practise, then see what the next visit could change. I work in audit, where a number needs supporting evidence. Here, that became a design question: when should the mirror hand movement back to each finger?

What it does

The working hand is the Lead. The weaker hand is the Understudy: Watch, Rehearse, Perform. The daily loop is measure, set assistance, exercise, compare next time.

Sit at your table and rest one hand. Understudy finds a surface and places a mat. A Hand Print measures open, fist and thumb reach over three trials. A felt cloth covers the weaker hand; the Lead's reflection moves above it. Each finger receives its own assistance setting. Watch follows the reflection; Rehearse blends it with the weaker hand's movement; Perform gives that movement the full role. Settings change only when measured range crosses an estimated error margin. The share describes the rendered blend.

One visit at the table: measure, mirror, per-finger shares, the coach's reasons, the next step, and a Visit from a second table. Frames of the app running in Meta's IWER emulator, over sample history.

A coach plans each routine from the measurements. It picks the movement lagging the Lead most, rotates in eight table exercises (stones to the bowl, turn the dial, spread, wave, flower, piano, thumb, rhythm) and explains every finger's setting in one sentence. The mirrored hand performs each exercise, even reaching its own reflected stones. Each session ends with Next visit: the range at which each finger takes its next step, and the last 14 days as dots, with no streaks.

Desk Break gives two working hands a two-minute routine at a desk. Phantom adds imagery for a missing hand; Morning Warm-up tracks stiffness. Later visit shows visit 22 over labelled sample history without touching personal records. Visit connects two tables by a four-digit code, so a relative or therapist sees your hands across the mat. Settings exports CSV or a one-page care-team summary, and erases local data.

Day one needs none of this. Onboarding asks which hand and which programme, then runs the first session; after that, one stone, Today, runs the whole routine. Progress, Programme, Settings, Later visit and Visit wait on Home until they are wanted.

How I built it

The daily loop. Assistance moves only when a finger's measured change exceeds its error margin; the bar shows what a share means on screen.

TypeScript and Meta’s Immersive Web SDK connect WebXR hand joints, passthrough, surface understanding and anchors. The real table sets the station’s position. Joint-local reflection and blending preserve hand geometry. IndexedDB stores personal history; a bundled offline cache supports returning to the routine. Prepared demonstrations use separate, labelled simulated records.

Challenges I ran into

Tracker noise can look like progress. I built procedural hands, synthetic movement curves and repeatability checks, then kept uncertainty visible. Very small attempts can be missed; the original 5° detection target is still unmet. In simulation, a 5° attempt is caught 89.5% of the time when the whole hand tries and 23% when one finger tries alone; a 10° single-finger attempt, 94%. That detector only decides whether the mirror helps during a repetition in Rehearse. How much each finger is helped comes from its measured range across visits, Watch always shows the help, and missing readings stay unavailable.

Accomplishments that I'm proud of

Every number comes from the repository: npm test (verification/software.json) and node tools/e2e-suite.mjs (verification/e2e-results.json), run on the desktop emulator.

The same hand sensors drive the mirror, controls and daily record. The app passes 178 unit tests, emulator journeys for every programme, a two-tab Visit and spoken commands, and 60 retained scanned-room placement checks. Pause, Resume, Stop, CSV export and confirmed erasure work. Phantom completes with either missing hand disconnected. A full production session completes offline.

What I learned

In the stated simulation, an index finger's change threshold is about 21°. Making that estimate visible is part of the experience design. A desktop geometry budget cannot establish headset frame rate or clinical benefit.

What's next for Understudy

Professional review and repeated measurements against an independent reference. March 2027 is a provisional release target.

Built With

• hand-tracking
• immersive-web-sdk
• indexeddb
• iwer
• passthrough
• peerjs
• pwa
• scene-understanding
• spatial-anchors
• three.js
• typescript
• vite
• vitest
• web-audio
• web-speech-api
• webrtc
• webxr