Inspiration

Rehab doesn’t stop when you leave the clinic. A lot of recovery happens at home, where it can be hard to know if you’re doing exercises correctly or making progress. We wanted to make that process a little easier with a simple tool that gives real-time feedback using a device people already own.

What it does

rupert is a home rehabilitation coach that runs in the browser.

  • Real-Time Biomechanical Tracking: Leverages on-device computer vision and pose estimation to extract skeletal landmarks and track joint kinematics across. Incorporates user-specific calibration, landmark smoothing, joint-angle analysis, repetition-state detection, and hold-duration tracking to quantify movement and deliver real-time corrective feedback.

  • EHR-Driven Rehabilitation Planning: Processes uploaded electronic health records (PDFs) through text extraction and LLM-assisted clinical information interpretation to generate personalized exercise plans. Uses a constrained JSON schema and a predefined exercise library to map extracted health information to supported rehabilitation routines while excluding exercises flagged as potentially unsuitable by relevant risk factors, such as elevated fall risk. Raw health-record documents are processed transiently and are not persisted; only the resulting exercise selections are retained.

  • Real-Time Fall Detection and Emergency Escalation: Analyzes continuous pose-estimation outputs using a rule-based detection pipeline to identify movement patterns consistent with potential falls. Upon detection, triggers an audible alarm and initiates emergency-contact notifications through automated voice calls and SMS using Twilio. Video frames remain on-device and are never transmitted as part of the alert workflow.

  • Interactive Musculoskeletal Visualization: Provides an interactive anatomical interface that maps rehabilitation exercises to their associated muscle groups. Dynamically highlights musculature targeted by the user's prescribed plan and surfaces relevant exercise history, enabling users to explore the relationship between individual movements, anatomical structures, and completed rehabilitation activity.

  • Client-Side Progress Analytics: Maintains a locally persisted rehabilitation history, including repetition counts, hold durations, completed sessions, active days, and longitudinal activity records. Presents these metrics through a progress dashboard while keeping session history within browser storage rather than requiring a remote database.

  • Gesture-Based, Hands-Free Interaction: Integrates MediaPipe Gesture Recognizer to support gesture-triggered exercise initiation, including thumbs-up recognition and raised-hand fallback detection. Applies hand-region cropping to improve gesture classification at a distance, enabling users to control rehabilitation sessions without physically interacting with the device between exercises.

How we built it

  • Next.js 16, React 19, and TypeScript, styled with Tailwind CSS v4.
  • MediaPipe Pose Landmarker (lite model, GPU with CPU fallback) for skeleton tracking, plus the MediaPipe Gesture Recognizer for the thumbs-up start. All camera processing happens in the browser; video never leaves the device.
  • Routines as state machines: Each routine is a class behind a shared interface (calibrate → update → complete → summary). The runner moves through ready → countdown → calibration → tracking → done.
  • Signal processing: Exponential moving-average smoothing, torso-length normalisation so thresholds work at different camera distances, and joint-angle measurements for rep detection.
  • Auto-framing: The displayed picture crops in to follow the user, while movement detection continues to run on the full frame.
  • Rule-based fall detection using the same pose stream, with alerts sent through Twilio.
  • Health-record plans: unpdf extracts PDF text, and the OpenAI API selects routines under a strict JSON schema that only permits exercises supported by the app.
  • Google sign-in with NextAuth, and an in-app help assistant built with the Vercel AI SDK and assistant-ui.
  • Testing: Headless test scripts simulate every routine end to end using synthetic poses, and the fall detector has its own test suite.

Challenges we ran into

  • Getting people in frame. Laptop cameras rarely capture the whole body, so we built a framing check that only enables Start when every joint needed for the exercise is visible, along with auto-framing.
  • Fall detection: Tuning rules to identify potential falls without triggering false alarms when someone sits or bends down.
  • Health data: Personalizing exercises using health records without storing them for the sake of maintaining user privacy.

What we learned

  • On-device pose estimation is fast enough for real-time coaching in a browser, but reliable tracking depends just as much on signal processing and user experience as on the model itself.
  • Privacy needs to be built in thoroughly: keep video on-device, avoid storing health records, and keep session history in the browser.

What's next

  • Clinician-assigned plans and shareable progress reports.
  • More routines and range-of-motion measurements over time.
  • Validating rep counting and fall detection with real patients and physiotherapists.

Built With

Share this project:

Updates

Submission history