RehabRobot

Inspiration

Rehabilitation works best when it adapts to the person doing it, but therapy time is limited and what happens between sessions is invisible to the therapist. We wanted to explore a take-home robot arm that patients can practice with every day, with session data reaching the therapist's dashboard. The goal is not to replace therapists. It is to save time for both patients and therapists.

What it does

Our prototype connects physical movement with personalized feedback:

• Assist-as-needed control: The patient starts the movement. If they stall, the SO-101 arm gradually adds assistance, within a torque cap set by the operator.
• Session tracking: Each session records range of motion, repetitions, assisted movement, and safety events.
• Remote upload: Session data can be uploaded from the website to the therapist dashboard, so practice between visits is visible without an in-person session.
• Two-camera perception: An external camera and a wrist-mounted camera stream live to a browser viewer.
• Therapist dashboard: Progress charts, personal baselines, goals, and milestones turn session records into something a therapist can review quickly.
• Grounded AI feedback: A local language model explains the measurements and answers questions, while deterministic Python code computes every number.
• Human-in-the-loop approval: The therapist reviews the next-session recommendation and approves it before the arm can use those settings.

This is a prototype. It does not establish clinical effectiveness.

How we built it

• Robot control: We calibrated SO-101 leader and follower arms and used the Feetech servo SDK to read joint position, speed, load, voltage, and temperature. A Python session controller counts repetitions, detects stalls, and adds assistance gradually. Safety checks cover excess speed, temperature, overshoot, communication failures, and mechanical end stops. A simulated motor bus lets us test the controller without hardware.
• Learning from demonstration: We used LeRobot to record 146 teleoperated demonstration episodes across five datasets (two 640x480 camera views plus six joint positions, at 20 FPS) and began training an Action Chunking Transformer policy.
• Dashboard and AI: A Streamlit dashboard sits on top of a single analysis.py module, so charts, suggestions, and AI explanations all share the same numbers. Ollama runs Llama 3.1 8B locally for summaries and Q&A. The model receives computed evidence and never calculates the statistics itself.
• Closed loop: JSON data contracts connect the arm controller, analytics, dashboard, and suggestion system. Suggestions are checked for staleness, mismatches, and prior use before they can reach the next session.
• Stack: Python, LeRobot, OpenCV, Streamlit, pandas, Ollama, PyTorch, and Git.

Challenges we ran into

• Demonstration quality: Inconsistent episode endings and changing camera placement hurt our training data. We fixed this by defining one movement per episode, fixing the camera position, building a camera alignment overlay, and reviewing and cleaning each batch.
• Safety: A motorized arm near a person needs conservative limits, so we capped assistance torque and added checks for speed, temperature, overshoot, and end stops. We still observed excess gripper squeezing during recording, which we are working on.
• Training setup: Our first ACT run on a local CPU was interrupted by a machine restart after 2,000 of a planned 20,000 steps, so we documented a GPU training path for the next run.
• Keeping the AI honest: We designed the language model to only explain numbers that Python computes, and the numerical charts still work when the model is offline.

Accomplishments that we're proud of

• A working rule-based controller, with three real rehabilitation sessions recorded on hardware.
• 146 demonstration episodes recorded and cleaned for imitation learning.
• Remote upload from the website to the dashboard, which moves us toward take-home practice.
• A dashboard that keeps measurement deterministic and uses AI only for explanation.
• A therapist approval flow that works end to end in simulation.
• 23 automated feedback and personalization tests passing.

What we learned

• Imitation learning depends heavily on consistent, well-labeled demonstrations. Data quality matters as much as the model.
• Separating deterministic analytics from language-model explanation makes AI feedback much easier to trust.
• Agreeing on data contracts early let the robot, analytics, and dashboard work together without constant breakage.
• Safety limits and therapist approval have to be part of the design from the start.

What's next for the project

• Finish GPU training and evaluate learned policies on the real arm.
• Validate dashboard-approved settings on hardware.
• Tune protective limits for safe home use and test the full take-home workflow.

Built With

• freetechsdk
• git
• lerobot
• llama
• ollama
• opencv
• pandas
• python
• pytorch
• streamlit