Mira | Devpost

Patient App Home Screen
Therapist Caseload Page
Watch Hi-fi

Inspiration

Currently, there is a physical therapy (PT) gap in America. About 38 million Americans living with chronic pain did not get outpatient physical or occupational therapy. Of 50 million with chronic pain, only 12 million got it. But why is that so?

From our research, we found that this could come from a number of reasons:

Time: Many people can't visit the PT office frequently and would rather do therapy at home.
Few PTs: There are not enough PTs for the people who need them.
Cost: After multiple PT visits, costs stack up and many are forced to pay out of pocket.
No Feedback: Home workouts can halt recovery if patients don't have a physical therapist giving feedback on their form.
Unequal access: Some areas don't have access to physical therapy at all.

We kept coming back to one person: Maya, 67, three weeks after a right knee replacement, living alone 45 minutes from her clinic. She does her squats every day. But with nobody watching, she has slowly started shifting her weight onto her good leg. Her squat depth is actually improving, so every individual session looks fine. The problem only shows up when you line up weeks of sessions and watch two curves cross: depth goes up while load on the injured leg goes down. By her next visit, she has trained a compensation pattern into her recovery.

Home-PT apps today log whether a patient says they exercised. We wanted something that measures how they moved, remembers it for months, and tells the therapist what nobody in the room would have seen.

What is Mira

Mira is an at-home physical therapy coach that connects webcam-based movement tracking, a patient app and a therapist dashboard.

Mira solves common issues that Americans come across when obtaining physical therapy.

Time: Patients can use Mira anytime of the day, when they are available.
Few PTs: Patients get automatically paired with a physical therapist.
Cost: Mira can be used without the large costs of appointments.
Feedback: On Mira, patients can easily get feedback from their PT, on what exercises to do and how to improve form.

What it does

The vision engine turns camera observations into a 3D skeleton in the mat's coordinate system. It measures joint angles, counts repetitions and checks exercise-specific form rules. The current rig uses one 720p webcam, and the stack also supports two-camera triangulation. A physics model estimates center of mass and left/right loading, with confidence checks that withhold unreliable load estimates.

The patient gets a coach display with Mira's mascot, spoken instructions, captions, countdowns and repetition counts. Holding a hand up starts a set when the engine is waiting for readiness. Leaving the mat during a set or crossing both arms pauses the exercise. These gestures use the camera rather than a microphone.

The iPhone app connects patients through a therapist-issued pairing code. It shows today's plan, a live skeleton, joint-angle and load measurements, coaching captions, session summaries and flagged moments. Patients can report pain and effort, view progress and message their care team. An optional Apple Watch companion adds workout heart rate, session controls and haptic cues through the iPhone.

The therapist dashboard brings together the caseload, session metrics, rep timelines, progress trends, messages, alerts and prescription editing. Therapists can compare two session dates with synchronized playback of stored movement data. The current dashboard displays angle labels and balance bars; an orbitable 3D mannequin is available in the laptop's vision preview.

Mira also includes several agents:

Care Copilot queries recovery data and drafts plan proposals that the therapist can approve, edit or dismiss.
Session Coach adjusts repetitions and rest within configured limits, using fatigue and heart-rate events.
Safety Guardian checks in after confirmed fall events, high pain or high heart rate. With live calling configured, it can escalate to an emergency contact and fall back to a therapist call.
Insight investigates longitudinal patterns and writes evidence-based notes for therapist review.
Data Guard checks that stored history and daily summaries remain consistent.

How we built it

Vision and physics: We used Python, OpenCV and MediaPipe Pose Landmarker to track 33 landmarks. ArUco mat markers establish camera geometry. In the one-camera path, calibrated floor geometry, MediaPipe's world landmarks and the patient's entered height guide a fitted body model. One Euro filtering and foot anchoring help stabilize the skeleton.

The loading model combines approximate body-segment masses into a center-of-mass estimate, applies a bounded inverted-pendulum correction and projects the result between the foot centers. This produces a camera-only estimate of loading, rather than a direct force measurement. Exercise thresholds and timed form rules live in JSON, with validated patient overrides.

Backend: A FastAPI server distributes pose, live state and events over WebSockets. The camera laptop runs a remote agent that sends movement data and receives session commands on the same connection. Camera capture starts after a session-start command. The backend manages sessions, coaching, summaries, messages, proposals and safety incidents.

AI and speech: Gemini generates coaching language, session summaries and therapist-facing drafts. ElevenLabs supplies cue audio, while Retell supports configured escalation calls. Prewritten safety prompts, cached cues and development fallbacks keep the application usable without every external service enabled. Movement checks are deterministic, and therapist plan changes remain reviewable proposals.

Patient interfaces: We built the iPhone and watchOS apps with Swift and SwiftUI, using WatchConnectivity for the phone/watch relay and HealthKit for workout heart rate. The phone also includes ARKit surface scanning and an illustrated setup guide. The LCD kiosk is a browser application designed for Raspberry Pi deployment, with a separately scoped device token.

Therapist interface: The JavaScript dashboard consumes the same REST and WebSocket contracts, connecting live measurements with session review, progress and care-team actions.

Tiger Data: Mira's longitudinal memory

We used PostgreSQL with TimescaleDB and Toolkit to turn movement streams into recovery history. The project supports a local Docker database and a configured Tiger Cloud service.

Pose frames, form events, alerts, heart rate and wrist motion use hypertables. A dedicated writer batches frames with PostgreSQL COPY every half-second, keeping database writes off the live event loop. A bounded 9,000-frame queue handles temporary connection interruptions.

A real-time continuous aggregate summarizes progress by patient, exercise and local day. Toolkit time-weighted averages account for uneven sampling, percentile summaries reduce sensitivity to isolated depth spikes, and LTTB queries provide smaller knee-angle series for review. The progress view translates left/right loading into loading on the patient's affected side.

In our seeded history, Maya's p95 squat depth rises from 61.3° to 85.5°, while estimated affected-leg load falls from 47.9% to 36.8%. These synthetic values demonstrate the pattern Mira is designed to surface.

The database also includes columnstore compression with before/after row fingerprints, restricted read-only access for AI queries and scheduled consistency checks. Emergency-contact columns are excluded from Copilot access.

The project's recorded mock-session and seeded-data sample reports approximately 41× faster progress queries through the aggregate, 2.74× compression, and zero raw-versus-summary mismatches across 799 summaries spanning 74 days.

Challenges we ran into

Reconstructing movement from one camera. Depth, occlusion and height accuracy affect the skeleton. Body initialization, confidence gating and filtering help avoid treating uncertain poses as reliable exercise measurements.

Keeping storage from interrupting coaching. We separated frame persistence into a buffered writer thread, with batching, reconnection and explicit overflow behavior. The project includes proxy-based database outage tests.

Making recent progress visible without hiding database problems. Real-time aggregates include new frames, while Data Guard checks the materialized summaries directly to detect missing refreshes.

Coordinating several interfaces. The vision engine, backend, phone, watch, kiosk and dashboard share REST and WebSocket contracts. Sample payloads and contract tests make those boundaries easier to verify.

Keeping AI actions controlled. Copilot queries pass through a SQL guard and a read-only database role. Session Coach clamps adjustments to configured limits, and therapists review plan proposals before applying them.

Accomplishments that we're proud of

Connecting camera-based tracking to coaching events, patient interfaces and persistent movement history.
Demonstrating a compensation pattern that a repetition count or depth measurement alone would miss.
Building patient and therapist workflows around the same session data, including summaries, check-ins, messaging and plan review.
Adding a watch companion and a mascot-based kiosk alongside the iPhone app.
Using time-series aggregates, compression verification and consistency checks to make longitudinal data practical to query and maintain.

What we learned

Movement feedback and recovery history answer different questions. A cue helps during a repetition; a trend helps explain what has changed over weeks.

One-camera tracking makes the rig simpler, but uncertainty needs to remain part of the measurement. Time-weighted averages, percentiles and confidence checks help us describe the data more carefully.

We also learned that useful AI needs clear boundaries: measured inputs, restricted tools, bounded adjustments and human review. Shared contracts and replayable data helped us build those boundaries across several devices.

What's next for Mira

Clinical validation with physical therapists on real patients, reporting detection rate and false alarms per exercise against labeled clean and cheated reps.
Rep similarity search: store each rep's angle curves as a pgvector embedding in Tiger so Mira can say "this looks like a weight-shift cheat" or "87% match to your therapist's reference rep."
Population-level insights across a clinic's caseload with more continuous aggregates, so therapists can see which exercises and cues actually speed up recovery.
More exercises, including forearm rotation using the Apple Watch for the twist cameras can't see.
Remote therapeutic monitoring documentation generated automatically from the data Mira already stores, so clinics can be reimbursed for home programs.
A simpler kit: one phone plus one webcam, set up once by the clinic.
Real emergency integration through Apple Watch fall detection or an emergency-services provider.