RehabMaze

Inspiration

Most physical therapy happens at home, and most patients quit. Clinics can measure a joint's range of motion in minutes, but at home nobody counts the reps, checks the form, or shows you you're improving, so motivation fades within weeks. We wanted rehab you strap on and play: a game that measures the exercise properly, adapts to you in real time, and gives your therapist real numbers instead of "I did them, mostly."

What it does

RehabMaze turns home rehab into a 3D garden maze game you steer with wearable motion sensors. You strap two IMUs on either side of a joint, forearm and hand or shin and foot, connected to an Arduino Nano over USB. You calibrate in under a minute: hold still, then make each slow movement the screen asks for, while an animated dot or lever shows exactly what to do. Then your wrist, ankle, elbow or knee tilts the maze to roll the marble, across 8 exercises. Each exercise gets a maze that matches it: up-and-down movements climb the Triangle Trail, and circles wind around the Circle Garden.

An AI coach reads your session every 5 seconds and adapts the game: faster when you're doing well, an easier target when you struggle, and a break when you tire. Each decision comes with a sentence of encouragement that uses your real numbers. The game also catches cheating, so twisting your arm instead of moving your wrist doesn't count.

When the session ends, you see your best and average reach, your hits and misses, and your angle over time. The session is saved to your garden automatically, earning Garden Points to spend in a Farmers Market on plants that appear in your maze. Your therapist gets a progress report with range of motion in degrees per direction, reps, smoothness, compensation, tremor and fatigue trends. No sensors on hand? A webcam fallback using MediaPipe pose tracking keeps the shoulder, elbow and knee exercises working on any laptop.

How we built it

Hardware

Two MPU6050 sensors share an Arduino Nano's I2C bus (addresses 0x68 and 0x69). Each sensor's onboard motion processor fuses its gyro and accelerometer into an orientation quaternion $ q_i = (w, x, y, z) $. The Nano streams both at 50 Hz, stamped with its own clock. The firmware saves the calibration on the board (ready about 2.6 s after plugging in), recovers a stuck I2C bus, reports a sensor whose wire comes loose, and corrects each gyro's drift live whenever it lies still.

The math that turns two sensors into a joint angle

1. Relative rotation. Each sensor sits on its body segment at an unknown, fixed mounting rotation $ m_0 $ and $ m_1 $. The joint's rotation is the second sensor relative to the first:

$$ q_{\text{joint}} = q_0^{\ast} \otimes q_1 = m_0^{\ast} \otimes q_{\text{anat}} \otimes m_1 $$

So turning your whole body doesn't change the angle; only the joint does.

2. Removing the strapping. A 3-second neutral hold (where $ q_{\text{anat}} = I $) measures $ q_N = m_0^{\ast} \otimes m_1 $. Every later reading becomes

$$ q_{\text{rel}} = q_N^{\ast} \otimes q_{\text{joint}} = m_1^{\ast} \otimes q_{\text{anat}} \otimes m_1 $$

the joint's true rotation, expressed in the sensor's frame.

3. Finding the joint's axes. During each slow calibration movement we take every sample's rotation vector $ \mathbf{r}_k = \theta_k \hat{\mathbf{n}}_k $ and find the principal axis of $ \sum_k \mathbf{r}_k \mathbf{r}_k^{\top} $. The movement passes only if that axis carries at least 80% of it:

$$ \hat{\mathbf{a}} = \arg\max_{\lVert \mathbf{a} \rVert = 1} \sum_k (\mathbf{r}_k \cdot \mathbf{a})^2, \qquad \frac{\lambda_{\max}}{\sum \lambda} \ge 0.8 $$

4. Separating movement from cheating (swing–twist). We split $ q_{\text{rel}} = (w, \mathbf{v}) $ into a twist about the limb's own axis $ \hat{\mathbf{a}} $ and the swing that's left:

$$ q_{\text{twist}} = \frac{\big(w, (\mathbf{v} \cdot \hat{\mathbf{a}}) \hat{\mathbf{a}}\big)}{\big\lVert \big(w, (\mathbf{v} \cdot \hat{\mathbf{a}}) \hat{\mathbf{a}}\big) \big\rVert}, \qquad q_{\text{swing}} = q_{\text{rel}} \otimes q_{\text{twist}}^{\ast} $$

The swing splits into flexion and deviation angles, and the twist angle $ 2\arccos(w_{\text{twist}}) $ is the compensation measure: past 40° in a circle, the board holds level.

5. From angle to game control. Each direction is normalised to the patient's own calibrated range, and a rep is a hit at 90% of it:

$$ c = \operatorname{clamp}\left(\frac{\theta - \theta_{\text{rest}}}{\theta_{\max} - \theta_{\text{rest}}}, 0, 1\right), \qquad \text{hit} \iff c \ge 0.9 $$

For circles, a joint reaches further in some directions than others, so we fit the patient's loop with a 4-harmonic Fourier boundary and stretch it onto a round circle:

$$ E(\varphi) = a_0 + \sum_{k=1}^{4} \big(a_k \cos k\varphi + b_k \sin k\varphi\big), \qquad \text{effort} = \min\left(1, \frac{\lVert \mathbf{p} \rVert}{0.9 E(\varphi)}\right) $$

Frontend

React, TypeScript and Vite. The browser reads the board directly over Web Serial, with no driver or app to install. Board timestamps are mapped to the page clock with a minimum-delay estimator, $ \Delta = \min_k \big(t^{\text{arrive}}_k - t^{\text{board}}_k\big) $, so USB jitter doesn't distort speeds. Angles drive a three.js tilting maze with marble physics, and a rep detector turns each reach (or each full turn, for circles) into a hit or miss. Range of motion, reps, SPARC smoothness, tremor (4–12 Hz band power) and a fatigue trend are computed live for the therapist. The webcam fallback runs Google's MediaPipe Pose Landmarker offline in a Web Worker on the GPU, smoothed with a One Euro filter.

Backend

A FastAPI server holds a WebSocket per play session. Every 5 seconds the browser sends validated telemetry (angle, speed, fatigue, hits and misses), and the server asks pt_coach, a Fetch.ai uAgent registered on Agentverse as a mailbox agent, so any agent on the network can query it. Fixed rules answer instantly, then Claude writes the sentence within an 8-second budget. If the agent is down, the backend runs the same rules itself. The garden side is a SQLite-backed REST API for session codes, therapy sessions, a points ledger that never double-counts, the Farmers Market and the progress report. Both halves are tested: 144 frontend and 66 backend tests.

Challenges we ran into

Movements slid back 20–30° after calibration. The sensor library calibrated gravity to 2 g instead of 1 g. The motion processor then balances a true tilt $ \theta $ against that extra 1 g on its Z axis, and settles at

$$ \tan\theta^{\prime} = \frac{\sin\theta}{\cos\theta + 1} = \tan\frac{\theta}{2} \quad\Longrightarrow\quad \theta^{\prime} = \frac{\theta}{2} $$

so every tilt read as half of itself. We wrote our own gravity correction and verified it reads 1.00 g.

Gyros under-read rotation by 7–8%. The library started the motion processor at 500 Hz, but its firmware is built for 200 Hz. A turntable test (four full turns read 1337° instead of 1440°) found it; after the fix, rotation is accurate to within 1–2%.

Warm sensors drifted up to 0.9°/s. Whenever a sensor is still for a second (under 3°/s spread), the firmware takes half of its measured bias out of the gyro's offset registers. Since a raw count is 1/16.4 °/s and an offset count is 1/32.8 °/s, the update is simply

$$ o_{n+1} = o_n - \bar{g}_n, \qquad b_{n+1} = \tfrac{1}{2} b_n $$

which halves the bias every still second: 0.9°/s → 0.00°/s within 30 s.

Two sensors without a compass drift apart, about 150° in three minutes of circling. A wrist can't twist about its own length, so any twist $ \tau $ the sensors report is drift. We turn sensor 1 back about the vertical by a heading $ \delta $, by gradient descent on $ \tau^2 $ while the joint moves:

$$ q_{\text{rel}}(\delta) = q_0^{\ast} \otimes R_z(\delta) \otimes q_1, \qquad \delta \leftarrow \delta - k \tau(\delta) \frac{\partial \tau}{\partial \delta} $$

The rate is capped at 5°/s, so a real compensation still shows. In simulation this undoes 119° of a 120° drift.

The game can't wait for an LLM. Claude takes 2–4 seconds, so each decision is split in two: the rules adjust the game instantly, and Claude's sentence replaces the stand-in text when it arrives. The Agentverse mailbox also rejected our agent until we traced it to the laptop's clock running 25 seconds fast.

Two apps, one record. We had built the rehab game and the garden as separate apps with separate designs. We merged them into one look, one backend and one record of every session.

When something "felt off", we built a recorder that logs every sensor reading during play and replayed real sessions offline. That's how we found nearly every bug.

Accomplishments that we're proud of

We made cheap hobby IMUs accurate enough for rehab: 1–2% rotation error, with drift brought to zero. Calibration works however the sensors are strapped on, guided by an animation, and the game knows the difference between real movement and compensation. The coach is a real AI agent on the Fetch.ai network: it changes the game live, and another agent got an answer through its Agentverse mailbox in about 7 seconds. And every session flows end to end, from sensor to game to coach to garden to a therapist-ready report.

What we learned

Sensor fusion is all about the errors you don't expect: library defaults, warm-up drift, and wrists that don't move like textbook hinges. Recording real data beats guessing every time. On the software side, letting rules make the decisions and the LLM do the talking kept the game instant, predictable and still human. And rewards should be for showing up, not for performance; points in RehabMaze never judge how well a movement went.

What's next for RehabMaze

Next come wireless ESP32 sensors so patients aren't tethered to a laptop, and a therapist dashboard across patients with prescriptions, trends, and alerts when someone stops exercising. We also want more joints (shoulder rotation, hip and neck), a coach that remembers each patient's patterns over weeks, and a pilot with physical therapists comparing our numbers against a goniometer.