-
-
Understudy gives each finger its own assistance setting in a seated mirror exercise; this is the real app in Meta IWER.
-
The synthetic tracker model gives index range a 20.8° change margin; repeated measurements guide each finger’s assistance.
-
Hand Print measures three movements over three trials and shows an estimated error margin; the hands here are simulated.
-
Watch, Rehearse and Perform use 0%, sample-derived and 100% own-movement shares; these describe the rendered blend.
-
The coach explains each finger’s setting from the same measurements and margins that set assistance; history is simulated.
-
Next visit shows each finger’s next change threshold and 14 days of visits; this prepared sample keeps personal history separate.
-
Progress places an earlier pose beside the latest and compares change with estimated error; this is simulated history.
-
Desk Break runs spread, wave, dial and stones for two working hands at a desk; this capture uses simulated hands.
-
Pause freezes the routine while Resume and Stop remain within reach; this is the real paused app in Meta IWER.
-
192 unit tests, four input journeys and 60 room-placement checks cover software and emulation.
-
Meta hand tracking drives the mirror, measurements and controls; local records feed assistance and the next visit.
-
Meta IWER’s actual developer controls around the deployed app show the emulator used for these captures.
About the project
Inspiration
Understudy is a seated daily mirror exercise for someone practising with one weaker hand. Meta hand tracking drives the reflection, measures movement and operates the controls.
A weaker hand is common after a stroke: upper-limb weakness was recorded in 77.4% of patients with a first-ever stroke in the South London Stroke Register (Lawrence et al., 2001). A Cochrane review of 62 studies with 1,982 participants found moderate-quality evidence that mirror therapy improves motor function (Thieme et al., 2018).
The moment I designed for is a morning at the same table: measure, practise, then see what the next visit could change. I work in technology audit, where a number needs supporting evidence. Here, that became a design question: when should the mirror hand movement back to each finger?
What it does
Rest a working hand to place the station. Hand Print measures opening, fist and thumb reach over three trials. A virtual cloth covers the weaker hand while the working hand leads its reflection.
Each finger has its own assistance setting. Watch follows the reflection. Rehearse blends it with tracked own movement. Perform follows own movement. Repeated measurements change assistance only beyond an estimated error margin.
One visit at the table: measure, mirror, per-finger shares, the coach's reasons, the next step, and a Visit from a second table. Frames of the app running in Meta's IWER emulator, over sample history.
A deterministic coach chooses from eight table exercises and explains each setting. Progress places an earlier hand pose beside the latest. Next visit shows each finger’s next threshold and the last 14 days as dots. Together, they give the daily routine a visible reason to continue.
Desk Break offers a short routine for two working hands at a desk. Phantom and Morning Warm-up serve other hand routines. Visit connects two tables by a four-digit code. CSV and a care-team summary make the local record shareable.
How I built it
TypeScript and Meta’s Immersive Web SDK connect WebXR hand joints, passthrough, surface understanding and anchors. The real table sets the station’s position. Joint-local reflection and blending preserve hand geometry. IndexedDB stores personal history; a bundled offline cache supports returning to the routine. Prepared demonstrations use separate, labelled simulated records.
Each session asks Quest for 72 Hz, the lowest Quest 3 / 3S rate above 60 fps, and renders both eyes through the SDK’s multiview stereo. Every shader compiles while the table is being found, so no screen pauses the first time it appears. Settings → Device check reports the session’s own frame rate.
The daily loop. Assistance moves only when a finger's measured change exceeds its error margin; the bar shows what a share means on screen.
Challenges I ran into
Tracker noise can resemble change. In the stated synthetic model, the index-range change margin is 20.8°. I made that uncertainty visible and kept missing measurements unavailable. Small attempts can be missed in Rehearse.
A headset compiles each shader the first time something uses it, and that frame waits. With the CPU slowed four times, a complete session in the emulator showed seven such pauses, the longest 306 ms. All 20 programs now compile during the first frames, a few milliseconds at a time; none compiles during the routine, and the longest frame fell to 66 ms.
Accomplishments that I’m proud of
All 180 unit tests pass. Four complete emulator journeys exercise right-hand poke, left-hand poke, dwell and head gaze. All 60 room-placement cases pass across five scanned rooms. A complete later session compiles no shader after it starts. The deployed later-visit and Desk Break routes run through the actual application with simulated hands.
Every number comes from the repository: npm test (verification/software.json) and node tools/e2e-suite.mjs (verification/e2e-results.json), run in Meta IWER.
What I learned
The useful feedback is specific: what changed, why assistance stayed or moved, and what the next visit could change.
What’s next for Understudy
Observed onboarding with intended users and an independent measurement review come next. March 2027 is the provisional launch target.
Built With
- hand-tracking
- immersive-web-sdk
- indexeddb
- iwer
- passthrough
- peerjs
- pwa
- scene-understanding
- spatial-anchors
- three.js
- typescript
- vite
- vitest
- web-audio
- web-speech-api
- webrtc
- webxr
