HandyAndy

Inspiration:

It started with a real situation: A friend fractured their hand and simply couldn't get in front of a doctor. The appointment was days away, and in the meantime, they were stuck with a swollen, painful hand and no idea what to actually do, which movements were safe, which exercises would help it heal, and which signs meant "stop, this needs a professional now." The advice that existed was a generic paper sheet of tiny hand photos that didn't tell them whether their hand was doing the movement right.

That gap —> between the injury and the appointment —> is where recovery usually stalls. HandyAndy was built to fill it.

What it does:

HandyAndy turns a doctor's note into a guided, at-home hand-rehab session, through three connected features:

Doctor's description → a personalised plan. You paste the clinician's note (e.g. "right index PIP stiff, gentle flexion, hold 5s"). HandyAndy first runs a safety scan for red-flag words (numbness, cold/blue finger, infection, severe pain…) and, if it finds any, blocks the exercises and tells you to call 111 instead. Otherwise it extracts a structured plan- which side, finger, joint, pain level, and the recommended approved exercise. 3D exercise guide. That plan drives an interactive 3D hand model (exported from Blender) that animates the exact prescribed movement on a loop, so you can see what "gentle flexion and hold" actually looks like, from any angle. Real-time exercise coach. You then practise in front of your phone camera, where on-device hand tracking watches your form live — checking reach, hold time, joint shape and whether you're compensating — scores each rep, and coaches you out loud. The camera feed never leaves your device. So it takes you all the way from "what did the doctor mean?" to "am I doing this correctly right now?"

How we built it:

The app is a single Next.js (React + TypeScript). The dashboard hosts the note analysis and the 3D viewer; the landing page tells the story. The note analysis is a deterministic, on-device extractor plus a red-flag safety scan, so it works with no API key and never invents a diagnosis or an exercise — it only structures what the clinician wrote. The 3D guide uses Three.js with a hand model we made and rigged in Blender, exported to glTF. We play just the relevant movement clip (the index–thumb pinch) on a smooth loop. The real-time coach is a separate web app built with Vite and Google MediaPipe hand-tracking, running entirely in the browser. We built it and embedded it inside the main app (served at /coach) so the whole thing runs as one product, launchable straight from the dashboard. Voice coaching speaks cues aloud (browser speech, with ElevenLabs voices), so you can keep your eyes on your hand.

Challenges we ran into:

Plenty. The biggest was the team itself: Aryan was the only actual developer, and the rest of the team were "vibe coders" — full of ideas but not writing the code — so getting things from concept into working, shippable code fell on one person and was genuinely hard to keep moving. On top of that, the medical side was complex: understanding hand anatomy (MCP/PIP/DIP joints), what's actually safe to recommend, and where the line is between "helpful guidance" and "don't give medical advice" took real effort to get right and to encode safely.

Accomplishments that we're proud of

Three features that, together, don't exist anywhere else — and they're all live in this project: turning a free-text doctor's note into a safety-checked plan, a 3D model that shows the exact prescribed movement, and a real-time camera coach that grades your form on-device. Most apps do one of these as a demo; we got all three working and wired into one flow.

What we learned:

A lot. Aryan learned Blender — modelling, rigging and exporting a hand that could be animated in the browser, which was completely new. Crabby learned ML — getting real-time hand-tracking and form analysis working from camera input. And as a team we learned how to take a messy real-world problem and a non-traditional team and still ship something that runs end to end.

What's next for HandyAndy

More of the body. The same loop — note → 3D guide → live camera coach — extends naturally beyond the hand to wrists, elbows, shoulders, knees and ankles. The goal is to make HandyAndy a recovery companion for any rehab plan, for that gap when you can't get to the clinic yet.