Flexion

Home hand exercises happen blind. Flexion gives them eyes: Alexa coaches a 2-minute morning check while a plain webcam measures every finger's range of motion.

Comment

Inspiration

This started with my father's fractured arm. A hand often comes out of a cast stiff, and getting the fingers moving again is mostly homework. The therapist sees the hand once a week, maybe less. The exercises happen at home, alone, and nobody sees how far the fingers really close. A stiff morning and real loss of motion feel the same from the inside, so the patient worries about the wrong days and the therapist hears "it felt a bit tight, I think".

A plain webcam can track 21 points on a hand. An Echo Show already sits in the kitchen and talks. I wanted both in one routine: a calm voice that runs the morning exercises and a camera that measures them. Plus one rule. A change only counts when it is bigger than the measurement's own noise.

What it does

You say "check my right hand". Amazon Nova 2 Sonic hears it and picks the MCP tool, and the Flexion coach opens on the Echo Show screen for a two-minute check:

Step What the coach says What the camera checks
Hello "When you're ready, hand up, palm to me." a hand in view, palm facing
Open ×3 "Open your hand wide, and hold it." fingers straight, held 1 s
Fist ×5 "Now a fist, thumb outside." the fist reading, with two form checks
10 reps "Open and close, ten times, slowly." each rep counted from the landmarks
Grip "Now pick up your phone." an object in the hand: phone, remote, book or mouse

The voice is Amazon Polly (Ruth), one line at a time. If your palm turns sideways it says "Turn your palm to face me." If your thumb hides inside the fist it says "Thumb across the outside." You can talk back: "wait", "slower", "it hurts" (it pauses and tells you to rest), and anything else goes to Nova 2 Sonic, which answers in two short sentences.

At the end the session is saved and you hear one of four verdicts: still building a baseline, inside your usual range, closed further, or looser. Then you can ask "how is my right hand doing?" for today against your baseline, or "make my weekly report" for a one-page PDF for your therapist with the trend, the noise band, pain flags and flare days. Every session also has its own PDF to download.

Alexa+ is not open to outside developers yet, so Flexion ships its own Echo Show host: a page that plays Alexa's part. It listens, asks Nova 2 Sonic which tool to call, calls the MCP server, and renders the coach as an MCP App in a sandboxed frame. The public site runs the same camera in a silent watch mode (both hands, dashed boxes around objects) and starts the coach only when you press Start therapy.

How we built it

The MCP server is Node 22 on AWS Lambda behind a Function URL (Lambda Web Adapter). It speaks MCP over Streamable HTTP with six tools: start_hand_check, get_progress, weekly_report, ping, plus save_session and identify_object, which only the coach app can call. The coach itself ships as an MCP App that the host renders in a sandboxed frame.

Layer Technology What it does
Voice Amazon Nova 2 Sonic (amazon.nova-2-sonic-v1:0, Bedrock bidirectional stream) picks the tool from free speech; answers questions mid-session
Coach voice Amazon Polly generative, Ruth 42 pre-rendered coach lines
Server AWS Lambda, Node 22, 2048 MB MCP tools, /api/voice, /api/identify
Storage Amazon DynamoDB, Amazon S3 sessions and routines; report PDFs and the vision model
Site AWS Amplify Hosting the public site with the live camera
Ops Amazon EventBridge, CloudWatch, AWS Budgets keep-warm ping every 4 minutes, logs, a 10 USD monthly budget alarm
Hand tracking MediaPipe HandLandmarker, GestureRecognizer, ObjectDetector 21 landmarks per hand, in the browser, on the GPU
Object check SigLIP base patch16-224 (q8) on onnxruntime-node names the object in the hand from 282 labels

Hand tracking runs in the browser at 30 fps in Chrome, under 12 ms a frame. The server never gets video: only the session log and, in the grip step, one 224 × 224 crop of the hand.

The measurement

The fist reading uses plain 2D webcam coordinates. Dividing by palm length cancels out how far the hand is from the camera:

$$r = \frac{\lVert p_{\text{index tip}} - p_{\text{index knuckle}} \rVert}{\lVert p_{\text{wrist}} - p_{\text{middle knuckle}} \rVert}$$

An open hand reads about 0.77 and a full fist about 0.18. Each held pose gives the median of a steady window, not a single frame.

A fist only counts if the palm faces the camera and the thumb sits outside:

$$\frac{w_{\text{knuckles}}}{L_{\text{palm}}} \ge 0.45 \qquad \frac{d_{\text{thumb tip, little knuckle}}}{L_{\text{palm}}} \ge 0.6$$

How big is noise?

Make the same fist ten times and the reading still wobbles. On the validation run (my right hand, 10 fists, both guards on) the standard deviation was \( s = 0.0226 \). The smallest real change at 95 % confidence is

$$\text{MDC95} = 1.96 \times \sqrt{2} \times s = 1.96 \times 1.414 \times 0.0226 \approx 0.063$$

Measured against the whole open-to-fist range, that is

$$\frac{\text{MDC95}}{r_{\text{open}} - r_{\text{fist}}} = \frac{0.0626}{0.766 - 0.183} \approx 0.107$$

or 10.7 % of the range. Flexion never calls anything inside that band a change.

Verdicts, progression and flares

After at least 3 earlier sessions the baseline \( b \) is their median, and today's change is

$$b = \operatorname{median}(r_1, \ldots, r_k), \quad k \ge 3 \qquad \delta = r_{\text{today}} - b$$

Inside the band (absolute change up to 0.063) it is a normal day. Past it, a negative change means the fist closed further and a positive one means it was looser.

The routine grows only inside the therapist's limit \( L \), by at most 10 %, and only after 3 completed pain-free sessions in a row and 7 days at the current level:

$$n_{\text{next}} = \min\left(L,\ \lfloor 1.1 \cdot n \rfloor\right)$$

A flare is two days in a row where the daily median reading is looser than the baseline by more than MDC95:

$$\tilde r_{d-1} > b + 0.063 \quad \text{and} \quad \tilde r_{d} > b + 0.063$$

The object check

The grip step first asks the in-browser detector. If it has no answer after 2 s, the coach sends the hand crop to /api/identify, where SigLIP ranks 282 labels by cosine similarity to the image. Its scores sit close together, so a fixed threshold is useless. Instead an accepted object passes when it is within 0.012 of the top label:

$$s_i = \cos(v, e_i) \qquad \text{pass} \iff \max_j s_j - s_{\text{accepted}} \le 0.012$$

Challenges we ran into

Bedrock said "Operation not allowed" to almost everything. The plan used Nova 2 Lite to read the therapist's paper exercise sheet. My account was still in verification, and every Converse and InvokeModel call failed, whatever the model or region (us-east-1, us-west-2, ap-south-1, ap-south-2). The one call that went through was InvokeModelWithBidirectionalStream. So Nova 2 Sonic became the voice layer, with my keyword rules kept only as a fallback. Told "my right hand feels stiff today, can you check it?", it called start_hand_check {"hand":"right"} by itself, and live it picks a tool in 0.86 to 0.95 s on the server. I moved object naming off Bedrock too. Amazon Rekognition got 1 of my 3 test objects right. Open-source SigLIP on the same Lambda matched all 5 test photos to their object and none of them to a cup.

The thumb decided whether the measurement worked at all. My first 16 fists gave an MDC95 of 29 % of the range, a clear fail. Three fists with the thumb tucked inside read 0.21 to 0.35, because the thumb stops the fingers from closing. Sideways fists read about 0.45 instead of 0.15. Leaving out the tucked-thumb fists brought it to 12 %, and a fresh run with both form checks on gave 10.7 %. The coach now speaks the same two fixes. I also dropped a 3D centimetre metric: its MDC95 was 57 to 100 % of the range.

The first coach voice scared my family. It started talking the moment the page opened and kept saying what it saw in your hand. It also cancelled itself at every new line, started a new line 250 ms after any banner change, and reset a 0.7 s hold on a single flickered frame, so it talked over itself and nobody could hold still long enough. I rebuilt it on 42 Polly clips, played one at a time and never cut off, with 1.2 s between lines, a 1 s hold that forgives a 0.4 s wobble, and 3 s to settle into each step. The camera now stays silent until you press Start therapy, and the voice never names what you hold.

Left and right were swapped. The MediaPipe docs say handedness assumes a mirrored image, so I flipped it. In my first real test, the PDF called "left hand" was my right hand. The red thread I wear on my right wrist sat on the image's right and was labelled LEFT, which showed the model was already right on unmirrored frames. I wrote a failing test, removed the flip and watched it pass.

The phone in my hand lost to my own body. In the grip step the detector drew "cell phone 58 %" around my hand, but the step never passed. I first loosened the hold rule, thinking a phone's box was too thin. The real cause was in holding(). It returned the first box holding the grip points, boxes were sorted by score, and the "person" box (me) always surrounds the hand and scores higher. Now the smallest qualifying box wins.

SigLIP scored differently on Lambda than on my PC. The scores moved just enough between the two CPU builds that a look-alike label which matched on the PC missed on Lambda. How a large photo is shrunk to 224 px changed the embeddings even more (0.85 to 0.97 alike). So the coach sends a fixed 224 × 224 crop, and the match uses the gap to the top label instead of an absolute score.

Accomplishments that we're proud of

  • A noise floor measured on a real hand (10.7 % of the range) and built into every verdict and every flare call.
  • Nova 2 Sonic choosing MCP tools from free speech, live, in under a second on the server.
  • An open-source vision model on Lambda: a warm object check takes 737 ms from India (p50), about 460 ms of it on Lambda.
  • 231 automated tests pass (server 188 with 1 skipped, site 43), covering the session engine, verdicts, progression, flares, the reports and the MCP tools.
  • The public site scores 100 for accessibility, best practices and SEO in Lighthouse on desktop and mobile, with full keyboard use and a designed reduced-motion path.
  • All of it open source under Apache 2.0.

What we learned

  • Measure the noise before you promise to detect change. MDC95 turned "better or worse" from a feeling into a rule.
  • Form cues are part of the measurement. A tucked thumb broke the numbers more than any camera did.
  • For a voice coach, timing beat voice quality. Never cutting itself off and forgiving a small wobble mattered more than the switch to Polly.
  • Keep a second path for every outside service. When Bedrock's text models stayed shut, Nova 2 Sonic took over the voice and an open-source model took over object naming.

What's next for Flexion

  • Read the therapist's paper sheet into a routine with its limits, with Nova 2 Lite, once Bedrock access opens on my account.
  • Run on real Alexa+ through the MCP Toolkit when it opens to developers, and meanwhile reach a physical Echo through a classic Alexa skill bridge.
  • Check the reading against a goniometer with a therapist, to turn the 2D ratio into degrees they already use.
  • More joints (wrist, thumb opposition), and Hindi and Hinglish now that Alexa+ has launched in India.

Built With

Share this project:

Updates

Submission history