CTRL+ALT+HEAL

CTRL+ALT+HEAL lets stroke survivors rehab at home: a camera compares the weak arm to the healthy one, an Apple Watch tracks daily use, and clinicians approve every plan change between visits.

Comment

Inspiration

One of us worked over the summer in a stroke-specialized emergency room and saw firsthand how a stroke changes a person's life. Sadly, once the patient leaves the hospital, recovery is far from over. Recovery depends on daily repetition at home, but no one on the care team can see whether it's happening or whether it's being done correctly. We built CTRL+ALT+HEAL to close that gap.

What it does

Stroke recovery depends on daily repetition at home, but the care team can't see what happens there. CTRL+ALT+HEAL closes that gap with one weekly loop: measure, review, approve, new plan.

  • Patient web app: guided 10-15 minute sessions with live camera feedback. The patient does each exercise with the healthy arm first, then the weak arm. Voice prompts count reps and warn about compensation ("Sit back against the chair"). The summary is one number: "Weak arm reached 64% of your healthy arm."
  • Care-team dashboard: flagged patients first, symmetry trends, alerts, and a rotatable 3D replay of any session that makes trunk lean and shoulder hike obvious. The app drafts the next plan and the clinician approves it with one click.
  • Apple Watch pipeline: heart rate, resting HR, and activity flow in daily, so the care team sees health trends between visits.
  • Weekly summary for the therapist: each week, Gemma reads the patient's recent exercise results and writes a short plain-language note: how symmetry and compensation are trending, whether the patient kept up with their sessions, and up to three suggestions for the therapist to consider. It's decision support only. It doesn't change plans or diagnose, and the clinician still approves every change.
  • Suggested plan, drafted by Gemma and approved by a clinician: after the weekly review, Gemma drafts the next plan from the patient's recent numbers: symmetry, compensation, pain, adherence and watch trends. It might step up to the next phase, adjust a plateaued plan, or focus on form when compensation is high. Each draft comes with a short reason, so the clinician sees why it was suggested and can approve it, edit it, or dismiss it. Nothing reaches the patient or the watch until a clinician approves it.

Clinicians approve every plan change. CTRL+ALT+HEAL never diagnoses, adjusts medication, or claims to detect a stroke, and the patient home screen always shows stroke warning signs with a call-911 link.

How we built it

  • No model training. Pretrained MediaPipe Pose and Hand Landmarkers run in the browser, so no video ever leaves the patient's home. Our code does the geometry on the landmarks.
  • Headline metric: $$ \text{Symmetry} = \frac{\text{Weak arm best}}{\text{Healthy arm best}} \times 100\% $$ This needs no clinical target numbers, because the healthy arm is the target.
  • Compensation detection: trunk lean and shoulder hike are computed from the shoulder-hip line and shoulder height. Angles are averaged over 5 frames and low-confidence joints are ignored.
  • Stack: React, Vite, TypeScript, and Tailwind front end. Three.js for the 3D replay. Recharts for trends. The Web Speech API for voice. Supabase (Postgres, REST API, Storage) as the single backend, deployed on Vercel.
  • Rules engine: one TypeScript function runs after each upload and flags plateaus, setbacks, compensation, missed sessions, high pain, and heart-rate or blood-pressure changes. Suggestions land as "suggested" until a clinician approves them.
  • Watch data: an iPhone Shortcut reads Apple Health and POSTs JSON straight to Supabase.
  • Gemma weekly summary: a Supabase Edge Function collects up to eight weeks of session results, pain scores, watch data and open alerts, and computes the trends in code. Gemma only receives those numbers, never video or free-text notes, and is told to use the trends exactly as given rather than recalculate them. A short, plain-text note is saved for the care team to review.
  • Gemma-drafted plans: a Supabase Edge Function sends Gemma only measured numbers (never video or free-text notes), and Gemma returns a draft plan with a reason. We save it as "suggested" with the data it was based on. Our rules engine still raises the alerts and sets the safety limits, and the clinician decides.

Challenges we ran into

  • Apple Health has no web API. Data never leaves the iPhone on its own, so we routed it through an iPhone Shortcut that POSTs to our backend.
  • Losing a teammate's laptop mid-hackathon. One laptop broke while we were cleaning it and became unusable, so we had to redistribute work and share machines.
  • The Apple Watch wouldn't sync to the computer. We couldn't rely on a direct connection, which pushed us to the Shortcut route and made our Watch data pipeline harder than planned.
  • Unreliable rep and angle tracking. Our first rep counter and joint-angle measurements were inconsistent, so we had to write more code and keep tuning it until the numbers were trustworthy enough to compare the weak arm against the healthy arm.

Accomplishments that we're proud of

  • A complete loop working end to end: session, upload, alert, suggested plan, clinician approval, and the patient's new plan.
  • Live weak-versus-healthy symmetry scoring, with live "sit back" compensation warnings.
  • A rotatable 3D session replay for the care team.
  • A privacy-first design where all camera processing happens in the browser.
  • A clear safety stance: clinicians decide, and the app never diagnoses.
  • Gemma writes the weekly summary and suggests the next plan from numbers the system already measured, every suggestion shows its reason, and nothing changes without a clinician's approval.

What we learned

  • Plan for hardware failure. Devices break and syncs fail, so we learned to keep work in a shared repo, avoid single points of failure, and have fallback data ready.
  • Platform limits shape the design. Apple's closed data ecosystem taught us to check what a platform allows before building on it, and to find workarounds like Shortcuts.
  • Measurement is the product. If the rep counter and angles aren't reliable, nothing downstream (symmetry scores, alerts, plan suggestions) means anything. Getting the geometry right took more iteration than the interface.
  • Interdisciplinary teams build better health tech. Our team ranged from business to nursing to philosophy, and computer science. The nurse grounded us in what stroke patients and clinicians actually need, which is why clinicians approve every plan change. The engineers made it work technically. The MBA kept the project focused on a real problem and a clear pitch. The philosophy major pushed us on ethics: what the app should never claim, and who is responsible for decisions. No single background could have built this alone.

What's next for CTRL+ALT+HEAL

Stroke is our starting point, not our limit. The same loop (measure at home, review, clinician approves, new plan) works for any condition where movement is the problem and the care team can't see what happens between visits. Next, we expand to other diseases that affect body motion and need physical therapy, with condition-specific changes:

  • Parkinson's disease: track movement size and rhythm, since movements often shrink and slow as a set continues. Add rhythm cues (beeps, voice) and log medication timing so the care team can compare "on" and "off" periods. Safety rules would center on balance and fall risk.
  • Multiple sclerosis: fatigue and symptom flares come and go, so sessions would be shorter with built-in rest breaks, and the app would flag sudden changes for clinician review instead of treating them as normal variation.
  • Spinal cord injury: exercises and limits are set by the injury level and adapted to wheelchair users, with blood pressure monitoring and strict safety stops.
  • Cerebral palsy: longer-term plans for children and teens, with caregiver involvement, playful exercise design, and growth-aware limits.
  • Post-surgery orthopedic recovery (knee, hip, shoulder): track range of motion, pain, and swelling against a surgeon's recovery timeline, with earlier alerts when progress stalls.

What stays the same across every condition: the clinician approves every plan change, the app never diagnoses, and all camera processing stays on the patient's device. What changes per condition is the exercise library, the metrics, the alert thresholds, and the safety rules, each set by that condition's specialists.

Beyond new conditions, our roadmap includes leg and walking rehab for stroke, a clinical trial to validate our measurements against therapist assessments, and a HIPAA-compliant deployment.

Built With

  • apple-health
  • apple-shortcuts
  • apple-watch
  • computer-vision
  • digital-health
  • gemma-api
  • github
  • healthcare
  • mediapipe
  • pose-estimation
  • postgresql
  • react
  • react-three-fiber
  • recharts
  • rest-api
  • supabase
  • tailwind
  • telerehabilitation
  • three.js
  • typescript
  • vercel
  • wearables
  • web-apeech-api
Share this project:

Updates

Submission history