Remotion

Inspiration

Rehabilitation continues long after a stroke survivor leaves the clinic. At home, small movements can take considerable effort, and differences between practice sessions can be difficult to notice or explain.

We built ReMotion to make that practice feel more supported and help therapists see what happened between appointments. Our goal was to connect everyday exercise with a movement history people can revisit and understand.

What it does

ReMotion is a webcam-based rehabilitation companion with voice-guided practice, movement tracking, and visual comparisons.

Users can practice finger spreading, seated knee extensions, and unweighted bicep curls. An ElevenLabs voice coach provides spoken guidance, while a virtual garden offers gentle encouragement for completing practice days.

Movement Echoes overlays silhouettes from previous recordings, making differences between movements visible.

For therapists, Ask ReMotion connects natural-language questions to recorded evidence. Asking which session had the greatest finger opening can identify the relevant recording and open it at the stored movement timestamp.

How we built it

We built the interface with React, TypeScript, Vite, and Tailwind CSS, using warm cream backgrounds, green accents, and clear controls.

MediaPipe provides hand and body landmarks for camera-estimated angles and repetition detection. Movement Echoes combines recorded video, extracted silhouettes, and landmark-based alignment without reshaping the recorded limbs.

Our Fastify backend supports authenticated access, PostgreSQL/TigerData movement storage, and Cloudflare R2 video storage.

We integrated ElevenLabs for conversational voice coaching and Gemini for question interpretation and session summaries. For movement searches, Gemini produces a validated intent; deterministic backend logic retrieves the actual measurements and recording references.

Challenges we ran into

Making comparisons meaningful required careful handling of camera viewpoints, body sides, calibration settings, and tracking quality. We added compatibility checks to avoid pooling measurements that should not be compared.

Connecting a result to its video was another challenge. Session identifiers, repetition indices, and timestamps had to stay aligned so the interface could open the correct evidence.

We also separated AI interpretation from measurement retrieval. Missing data produces an explicit empty result, and session-level measurements never receive invented timestamps.

Accomplishments that we're proud of

Connecting movement-history questions to actual recordings.
Building Movement Echoes to compare recorded movements visually.
Bringing voice guidance, tracking, and gentle encouragement into one experience.
Passing 141 automated tests, backend type checks, and production builds, with browser verification using team demonstration footage.

What we learned

We learned that useful feedback needs more than a number. People need to know what was measured, which recording supports it, and what its limitations are.

We also learned to design encouragement carefully. Our garden recognizes completed practice days without rewarding excessive repetitions or penalizing rest.

Throughout development, we kept an important distinction visible: webcam measurements are estimates, and recorded differences do not establish clinical recovery.

What's next for ReMotion

We plan to expand exercise coverage, improve tracking across home environments, and validate the complete cloud-connected experience. Clinical evaluation remains future work; our prototype currently focuses on supporting prescribed practice and making recorded movement easier to review.