Bend With Us

Your device becomes a physical therapist: measuring every exercise, coaching with a multi-lingual voice assistant, and sending your therapist real metrics. Video never leaves your device.

Comment

Banner

Inspiration

The inspiration behind our project began with a story close to our team. One of our teammates' grandfather suffered multiple strokes. The primary reason he recovered as well as he did was that a family member who was a professional massage therapist could rehab with him almost daily. Whenever he stayed with relatives who didn't know his routine, he returned noticeably weaker. Most people recovering from a stroke, surgery, or joint injury simply don't have a professional checking on them between clinical visits.

Only about 35% of physical therapy patients fully follow their home rehab plan, and adherence drops from roughly 64% early in treatment to just 23% by the end. Most of knee, hip, or shoulder rehab happens at home, alone, with a paper handout and no objective measurement between visits. A goniometer (the plastic protractor physical therapists use) costs a few dollars but requires a second person to read it.

We built Bend With Us to bridge the gap between the clinic and the living room.

What it does

Bend With Us is an AI physical therapy coach for knee, hip, shoulder, elbow, and wrist rehab.

For the patient: set your phone or laptop camera side-on. The app checks placement, framing, and lighting automatically before starting. As you exercise, local pose tracking measures your joint angle frame-by-frame, counts reps, and checks form in real time. A voice coach provides live audio feedback (ex. "too fast") in English or Spanish while you exercise. Audible expression of pain, like saying "it hurts," immediately halts the session and triggers a spoken pain check. Each week, patients receive a recap of their progress.

For the therapist: a caseload dashboard tracks adherence, patient milestones, pain check-ins, and red flags. There's a range-of-motion chart for every joint, a rep-by-rep replay of any session, and a live view of a session in progress (angles only, never video). Google Gemini can draft plan changes (progress, hold, or regress) that cite the patient's own numbers, but nothing reaches the patient until the therapist approves it.

Video never leaves the device. Only anonymized angles, reps, and pain scores are stored.

How we built it

Pose tracking in the browser: MediaPipe Pose runs on the device and creates body landmarks every frame.

The Stack:

  • Frontend: React 19, TypeScript, Vite, and Tailwind (compiled to iOS and Android via Capacitor for mobile). Recharts renders the therapist charts. Computer vision runs on the frontend via MediaPipe Pose Landmarker, calculating joint angles on-device.

  • Backend: FastAPI running on DigitalOcean. WebSockets power live patient sessions, while Server-Sent Events (SSE) update the therapist dashboard. Auth is handled with JWT and Google Sign-in. Patient and therapist access controls are enforced directly at the database level.

  • Google Gemini API: the rehab plan suggestion, weekly summary and recap, pain-check replies, and pulling the score and symptoms out of a spoken answer.

  • ElevenLabs: one coach voice in two languages. 130 cue clips are pre-generated with eleven_multilingual_v2, and live replies stream with eleven_flash_v2_5 so audio starts before synthesis finishes. Scribe handles speech-to-text for the pain check-in.

  • Tiger Data: every angle frame of every session goes into a hypertable. A real-time, continuous aggregate rolls sessions up per minute. time_bucket serves the replay and the daily trend, and columnstore compression shrinks finished sessions by about 11:1 on our data.

  • DigitalOcean: hosts the API and the web app on bendwith.us, a .us domain registered with GoDaddy Registry.

Challenges we ran into

Landmark points are jittery. To combat this issue, we added frame-smoothing algorithms and coordinated aspect-ratio corrections to make angle measurements clinically trustworthy.

2D only works side-on. A camera measures the angle projected onto its image plane, so a joint turned towards the lens reads incorrectly. Instead of hoping patients set up correctly, the app checks placement, framing, and lighting, only starting when all three pass.

Seated hip reps never ended. The seated hip exercise rests at about 86 degrees, not 0 degrees, and patients who leaned forward to lift and then sat that way got stuck mid-rep. The counter now learns each patient's rest angle, moves the rep thresholds with it, and ends a rep wherever the leg settles.

Voice latency. A coach that answers a second late feels broken. Fixed cues are pre-generated clips, and only the replies that must be live are streamed. Now, we have a voice coach with near-zero delay.

Accomplishments that we're proud of

  • A real clinical task done end to end: you move, it measures, and your therapist sees it.
  • Data a therapist can act on: live rep by rep angle measurements, historical patient charts, and AI-powered summaries.
  • Clinical Human-in-the-Loop AI: a plan-change loop where AI drafts, rules check, and a human decides.
  • Single Codebase: one codebase on the web, iPhone, and Android.
  • Pose Accuracy: a built-in accuracy harness (/pose-debug) that records held poses against a goniometer and reports mean absolute error and Bland–Altman limits of agreement.
  • Built-in Patient Privacy: video and personal data never leave the device.

What we learned

  • Choose the exercise to suit the camera. Seated, side-on movements give a webcam a fair chance at clinical-grade readings.
  • With AI in a healthcare workflow, the fallbacks and guardrails are mandatory. The model can draft, but deterministic safety rules and human oversight must govern patient care.
  • Voice changes user interactions. Timing needs to be fluid and the words being said matter.
  • Much of physical therapy is adherence. A patient who can see their progress will stick with their routine.

What's next for Bend With Us

  • Publish peer-reviewed accuracy data comparing our vision engine against traditional goniometer measurements across broad demographic groups.
  • Expand exercise offerings per joint, including routines utilizing multi-camera setup options.
  • Integrate directly with Electronic Health Record (EHR) systems to automatically synchronize therapist summaries and progress notes.
  • Partner with physical therapy clinics and healthcare providers for real-world pilot studies.

Built With

Share this project:

Updates

Submission history