TrueReach

Login Page
Physician Page
Physician View In Depth
User Page
Overhead View of AutoArm
Side View of AutoArm

Inspiration

Around 15 million people have a stroke every year, and most survivors come away with some loss of arm function. For many it lasts: more than half still have arm weakness a year later. Getting that function back takes a lot of repetitions, and most of those reps happen at home with no therapist watching.

The problem with unsupervised reps is compensation. When the affected arm is weak, patients get the hand to the target some other way: they lean the trunk forward, hike the shoulder, or twist the torso. The rep looks finished, but the arm didn't do the work. Practicing that pattern at home reinforces it and can push recovery back. In clinic, a therapist catches this with a hand on the shoulder. At home, nobody does.

What it does

TrueReach watches a patient do a prescribed reaching exercise at home, catches compensation as it happens, and sends the session to their therapist afterward.

How it fits into care:

In clinic: setup and calibration. The patient sits with their therapist, who prescribes the exercise, sets up the station, and gets the patient used to wearing the sleeve and using the software. The therapist adjusts the compensation thresholds for that specific patient.
At home: practice on their own schedule. The patient does their reps whenever works for them. If they compensate, the sleeve buzzes and the screen highlights the body part with the measured value (e.g. "Trunk lean 14°"). Every rep is scored: clean, trunk lean, shoulder hike, trunk rotation, incomplete reach, or uncertain when the camera can't see the body well enough to judge.
After the session: the therapist reviews. The recording and rep results upload to a physician portal. The therapist can replay the session with the skeleton overlay and see patients ranked by how much of their time they spent compensating. They step in only when something needs attention.

The patient gets more practice and more freedom, and the therapist keeps oversight without spending clinic time watching every rep.

It understands the exercise, not just angles. Following the Reaching Performance Scale for Stroke (RPSS), leaning toward a near target is flagged as compensation. Leaning toward a far target is expected, so it isn't flagged.

Why a sleeve and not just a webcam app? A patient in the middle of a reach can't watch a screen. A buzz on the arm or chest is feedback they feel without looking, which is the closest thing to a therapist's hand on the shoulder.

How we built it

Computer vision. MediaPipe Pose Landmarker runs in the browser on the laptop's built-in webcam, set up to the patient's affected side. For every frame we compute:

trunk lean (the shoulder-to-hip line against vertical)
shoulder hike (the drop in ear-to-shoulder distance, normalized by trunk length)
elbow angle

Each one is measured relative to a rest pose captured at the start of the session.

Hardware. An ESP32 reads three IMUs (upper arm, forearm, chest). A Madgwick filter on the ESP32 fuses accelerometer and gyro data into an orientation quaternion for each sensor at 100 Hz, and streams them over USB to the browser through the Web Serial API. The chest IMU gives trunk lean and trunk rotation, which a side-view camera can't see. The arm IMUs give elbow angle. Two vibration motors, one on the chest and one on the upper arm, give different buzz patterns for different compensations. We wrote a small serial protocol for:

streaming sensor data
zeroing gyro bias
driving motor patterns at different strengths

Thresholds from clinical data, not guesses. We used the Toronto Rehab Stroke Pose (TRSP) dataset, which has Kinect skeletons with expert per-frame compensation labels.

We fit each threshold on the healthy participants, picking the cutoff that best balances sensitivity and specificity.
We then tested on the stroke patients' forward-reach recordings, which weren't used for fitting.
Because TRSP is 3D and our webcam is a 2D side view, we computed the TRSP features in the side-view plane so they'd match what the camera sees.
Target placement (near at 1 cm, far at 30 cm from the table edge) follows the RPSS setup.

In real use, these would be starting defaults that the therapist tunes for each patient.

App.

Frontend: React + Vite + TypeScript, Tailwind, shadcn/ui, Recharts
Backend: FastAPI
Storage: Supabase (Postgres + Storage). Each session saves the webcam video plus a timeline of every sample (pose, features, flags, sensor data), so the physician replay stays in sync with the video.
Login: Auth0 gates the physician portal, and only accounts with the physician role get in.

From MVP to something deployable

What we demoed is an MVP, and the hardware on the table is a wired prototype. A deployable version would be much less intrusive: a simple sleeve and a small chest monitor, wireless (the ESP32 already has Bluetooth) and battery-powered, with no loose wires or breadboard. The software flow would stay the same.

To get there, we'd need to:

Validate on our own setup. Record data with our own camera and sleeve and measure real accuracy on it. Right now our thresholds come from Kinect data applied to a 2D webcam, so live accuracy is likely lower than on TRSP, and we haven't measured by how much.
Work with therapists on threshold tuning and on adding more exercises (the code is already generic over exercise profiles).
Lock down patient data properly. Today the login gates the UI, but the database itself isn't fully secured yet.

To be clear about what we don't claim: clinical validation, measured accuracy on our hardware, or replacing a therapist. The point is to extend the therapist's reach between appointments.

Challenges we ran into

Getting 3D Kinect training data to line up with a 2D side-view webcam
Our IMUs turned out to be MPU6500-compatible clones, so we had to drop the standard library and read the registers directly
Getting gyro drift under control well enough for a stable trunk angle
Building camera-first so the software never had to wait on hardware, then integrating the sleeve late without breaking the working demo

What we learned

How compensation is actually defined and scored clinically (TRSP, RPSS), that feedback a patient can feel matters more than another number on a screen, and how much planning it takes to run four people in parallel on one repo over a weekend.