About the project

Inspiration

Understudy is a seated daily mirror exercise for someone practising with one weaker hand. Meta hand tracking drives the reflection, measures movement and operates the controls.

A weaker hand is common after a stroke: upper-limb weakness was recorded in 77.4% of patients with a first-ever stroke in the South London Stroke Register (Lawrence et al., 2001). A Cochrane review of 62 studies with 1,982 participants found moderate-quality evidence that mirror therapy improves motor function (Thieme et al., 2018).

The moment I designed for is a morning at the same table: measure, practise, then see what the next visit could change. I work in technology audit, where a number needs supporting evidence. Here, that became a design question: when should the mirror hand movement back to each finger?

What it does

Rest a working hand to place the station. Hand Print measures opening, fist and thumb reach over three trials. A virtual cloth covers the weaker hand while the working hand leads its reflection.

Each finger has its own assistance setting. Watch follows the reflection. Rehearse blends it with tracked own movement. Perform follows own movement. Repeated measurements change assistance only beyond an estimated error margin.

A visit at the table, in six frames One visit at the table: measure, mirror, per-finger shares, the coach's reasons, the next step, and a Visit from a second table. Frames of the app running in Meta's IWER emulator, over sample history.

A deterministic coach chooses from eight table exercises and explains each setting. Progress places an earlier hand pose beside the latest. Next visit shows each finger’s next threshold and the last 14 days as dots. Together, they give the daily routine a visible reason to continue.

Desk Break offers a short routine for two working hands at a desk. Phantom and Morning Warm-up serve other hand routines. Visit connects two tables by a four-digit code. CSV and a care-team summary make the local record shareable.

How I built it

TypeScript and Meta’s Immersive Web SDK connect WebXR hand joints, passthrough, surface understanding and anchors. The real table sets the station’s position. Joint-local reflection and blending preserve hand geometry. IndexedDB stores personal history; a bundled offline cache supports returning to the routine. Prepared demonstrations use separate, labelled simulated records.

Each session asks Quest for 72 Hz, the lowest Quest 3 / 3S rate above 60 fps, and renders both eyes through the SDK’s multiview stereo. Every shader compiles while the table is being found, so no screen pauses the first time it appears. Settings → Device check reports the session’s own frame rate.

The daily loop and one finger's share The daily loop. Assistance moves only when a finger's measured change exceeds its error margin; the bar shows what a share means on screen.

Challenges I ran into

Tracker noise can resemble change. In the stated synthetic model, the index-range change margin is 20.8°. I made that uncertainty visible and kept missing measurements unavailable. Small attempts can be missed in Rehearse.

A headset compiles each shader the first time something uses it, and that frame waits. With the CPU slowed four times, a complete session in the emulator showed seven such pauses, the longest 306 ms. All 20 programs now compile during the first frames, a few milliseconds at a time; none compiles during the routine, and the longest frame fell to 66 ms.

Accomplishments that I’m proud of

All 180 unit tests pass. Four complete emulator journeys exercise right-hand poke, left-hand poke, dwell and head gaze. All 60 room-placement cases pass across five scanned rooms. A complete later session compiles no shader after it starts. The deployed later-visit and Desk Break routes run through the actual application with simulated hands.

What was verified Every number comes from the repository: npm test (verification/software.json) and node tools/e2e-suite.mjs (verification/e2e-results.json), run in Meta IWER.

What I learned

The useful feedback is specific: what changed, why assistance stayed or moved, and what the next visit could change.

What’s next for Understudy

Observed onboarding with intended users and an independent measurement review come next. March 2027 is the provisional launch target.

Built With

  • hand-tracking
  • immersive-web-sdk
  • indexeddb
  • iwer
  • passthrough
  • peerjs
  • pwa
  • scene-understanding
  • spatial-anchors
  • three.js
  • typescript
  • vite
  • vitest
  • web-audio
  • web-speech-api
  • webrtc
  • webxr
Share this project:

Updates

Submission history