Baymax

HIPAA compliant physical therapy AI assistant.

Comment

Baymax: a physical therapy robot that shows you the move, watches you do it, and stops when it hurts

Inspiration

Most people skip their physical therapy exercises. Nobody is watching, nobody corrects the form, and if something hurts, the therapist hears about it two weeks later at the next appointment. We wanted something in the room that would show you the move, watch you do it, and tell your therapist what happened. It's called Baymax because a small friendly robot whose only job is to look after you was the obvious joke, and we couldn't think of a better one.

What it does

Baymax runs on a Hiwonder TonyPi Pro, a small humanoid with a Raspberry Pi 5 inside.

It starts by looking for you. It sweeps its head around the room, follows faces, and puts a smile on its LED matrix when it finds one. When you wave or say hi, it walks over, stopping about 50 cm away. It does a short fall-safety demo (it lies down on purpose, tells you to say "help" if you ever get hurt, then stands back up), takes your temperature with the sensor on its right hand, and asks how you're feeling.

Then comes the workout. For each exercise it says the name, does the move itself, and watches you through its camera while you do it. It counts reps, checks your form against joint angles, and gives feedback after each set. It asks how the set felt and writes down the answer.

If you say "ow", "stop", or "that hurts", the servos freeze right there. The robot stays put until you tell it you're okay. If you say "help", it ends the session, tells you to call 911, and marks the notes as an emergency.

Afterward the therapist gets a dashboard: a score for the session, the reasons behind it in plain words, and a chart for each reason. They can replay the whole session on a 3D copy of the robot and ask an AI coach questions about it.

How we built it

The robot is just a body. The TonyPi runs one HTTP API that moves servos, speaks, reads sensors, and streams its camera. Everything else runs on an ASUS GX10 sitting on the same network. We split it this way so the robot could stay simple and every smart piece could be tested without it.

Seeing. The camera streams to the GX10 as H.264 over UDP. YOLO pose tracks everyone in view, spots waves, and estimates distance from how big a person looks. Joint angles drive the rep counter and the form checks. MediaPipe is there as an alternative, though it usually finds only one person.

Hearing. The microphone streams as Opus over UDP. Silero VAD finds where each utterance starts and stops, and Whisper transcribes it on the GPU.

Deciding. We built two session runners and kept both. workout/ is a plain state machine (ready check, demo, mirror, set review) with safety interrupts allowed from any state. pt_agent/ hands the same job to an LLM, Qwen 3.6 35B on Ollama or any Claude model, which picks each step from a session plan using tools like wait_for_patient, approach_patient, and take_temperature. The state machine is predictable. The agent is friendlier to talk to. We still don't know which one patients would prefer.

Reviewing. The dashboard is FastAPI with a React front end. Besides session review, it has an animation library, a pose studio, a recorder where you let the arms go limp and move them by hand, and a calibration screen. There's also a motion generator: describe a move in words and the model on the GX10 writes the keyframes, which took about 12 seconds in our tests. A URDF digital twin in three.js shows the real robot's servo readings next to what we commanded, so drift and stalled servos are easy to see.

Keeping it safe in code. Every motion goes through calibration limits in the dashboard, and then the robot API clamps it again. Stop is on every screen. Anything arriving over the LAN is validated. Pain words are handled by the thread that hears them, so they never wait for the model.

Nearly every entry point takes --dry-run, which swaps in a fake robot. That's how most of this got built: on laptops, with a webcam standing in for the robot's camera.

Challenges we ran into

The servos run hot. After about 25 minutes of standing they reached 74 °C, and they cut out at 85 °C. We added a temperature check and now plan demos around cool-down breaks.

Speech stalled for about 30 seconds on every other sentence, because PipeWire was holding the sound card. Routing audio through PulseAudio fixed it, and it took us far longer to find than to fix.

Model thinking was too slow. With it on, a reply could take half a minute, which is a long time to stand in front of a robot waiting for it to answer. We turned thinking off by default for the Ollama models and replies dropped to a few seconds.

The motion generator's model didn't follow our keyframe schema exactly. We only found that by pointing it at the real model on the GX10, since the offline mock always behaved. We added validation and repair for it.

And everything is shared. Only one program can hold the camera, a UDP port, or the servo bus at a time, and teammates worked on the same two machines. Half our "bugs" were someone else's leftover gst-launch process.

Accomplishments that we're proud of

The whole loop works: wave at a robot, get coached through a set, and read a scored report afterward. It touches vision, speech, two kinds of decision-making, servo control, and a web app.

The dashboard passes 15 desktop and 5 phone flows in a real browser, and the Python suite has 196 passing tests, all run against the fake robot.

We're also proud that the pain stop doesn't depend on a language model behaving.

What we learned

Don't let the model near the stop button. The LLM decides what to say. Stopping is ordinary code on the thread that heard the word.

Test against the real model early. The mock was always polite.

What's next

Pose tracking in live dashboard sessions is still simulated in some modes, and knee and trunk angles need a better view than our arm-focused estimator gives. Servo cool-down needs to be scheduled instead of remembered. Approach distance and head direction still need tuning on the physical robot.

The question we can't answer yet is whether any of this actually helps. We haven't run it with a real patient, so we don't know whether form feedback from a 40 cm robot makes anyone do their exercises more, or just makes them laugh and quit. We'd like to find out.

Built with: Python, FastAPI, React, TypeScript, three.js, YOLO pose, MediaPipe, Whisper, Silero VAD, Ollama (Qwen), Claude, LangChain, GStreamer, Raspberry Pi 5, Hiwonder TonyPi Pro, ASUS GX10

Built With

Share this project:

Updates

Submission history