RePlay

bird's eye view of our device!
mario kart game with our hand controlling it
steering the mario kart with my hands !!
our progressive mario game feedback using Baseten + OpenAI :)
our sentry beedie for our hardware errors!
zelda game with us being able to squeeze
squeeze mechanism to release bow + arrow
paintball combat 3D game!
money shot
a final close up :)

🎯 Inspiration

Physical therapy after a stroke or mobility loss is long, repetitive, and boring. When exercises feel like a chore, motivation drops and recovery slows.

RePlay turns prescribed upper body exercises into arcade games, so the reps a therapist asks for are the reps a patient wants to do.

🎮 What it does

Three motion controlled games driven by a wearable sensor (Mario race cart, Legends of Zelda Bow & Arrow fight, Paintball machine), with a camera acting as assessor rather than controller.

Every input is a prescribed movement: Forearm rotation steers a kart, a grip squeeze draws a bow/releasing it looses the arrow, and a lateral sweep past a calibrated zero swings a sword.
Control and assessment are separate: The controller drives the game. A webcam tracks the hand through MediaPipe and records its path alongside the driven path, scored independently against an ideal line: mean and worst deviation, signed bias, path efficiency ratio, and first third versus last third error to surface fatigue.
Difficulty is proposed, then clamped: After each run, an LLM reads the metrics and proposes changes to lane spread and pickup window. Proposals are clamped to TUNING_BOUNDS with a max step per session, so the model tunes but cannot push a patient past a safe range.
Therapy notes compile into games: Upload a therapist's PDF and an agent extracts the prescription, resolves conflicts, validates it, and emits a playable game spec for a person's individual rehab needs.

🛠️ How we built it

Hardware: ESP32 with an MPU6050 IMU and a force sensitive resistor, in a rechargeable wearable. Firmware runs a complementary filter, normalises each axis against a per patient calibrated range of motion, and streams 8 CSV channels (roll,pitch,yaw,steer,moveX,moveY,rawFsr,squeeze) at 115200 baud over Bluetooth Classic. Single character commands (c, r, l, f, m) recalibrate neutral, rotation limits, and grip baselines live.

Input: The browser reads the stream through the Web Serial API, with a parser that accepts both firmware revisions (5 field and 8 field) and auto detects whether grip arrives as 0 to 1 or as a 0 to 255 byte. A gesture layer maps signals to actions: bow release fires on a sharp pressure drop scaled against a realistic grip maximum, and slash fires on displacement past a calibrated zero, re-zeroing the device between reps.

Games: Next.js 16 and React 19. Two games render on Canvas 2D + the first-person paintball game renders on Three.js and WebGL.

Vision: MediaPipe Hand Landmarker maps palm position into the same road units as the car, so both paths are directly comparable.

Baseten's Model APIs: Serve zai-org/GLM-5.3, which reads each session's movement metrics and writes the patient's coaching note plus a proposed difficulty adjustment. We request structured JSON so the response drops straight into the game, with retries, timeouts, and an OpenAI fallback so a slow model never blocks a session.

OpenAI API:

gpt-4.1-mini runs the compiler as a tool loop, not a single prompt. Five tools: extract_exercise_claims, find_conflicts, validate_plan, ask_clarification, generate_game_spec. Every tool call is parsed through a Zod schema before it touches state, so a malformed plan is rejected rather than rendered. The same model backs session coaching when Baseten is unavailable.

For example, a therapy note like:

“Right shoulder flexion, 2 sets of 8. Avoid trunk lean. No overhead reaching.”

can compile into a rock-climbing game with eight holds, where the patient’s prescribed shoulder movement controls their reach. The agent preserves the prescribed repetitions and movement constraints while translating the exercise into gameplay.

Codex helped us move from isolated hardware demos to a unified input system. It helped us refactor two incompatible wearable data formats into one normalized browser input layer, allowing every game to consume the same calibrated movement signals.

🧠 Custom Game Generation Agent

Therapy instructions can be messy, but even when they're clear, repeating the same prescribed movements every day gets boring.

RePlay’s Therapy Game Compiler is an interactive agent that turns this reality into a personalized playable game.

Our extraction pipeline starts with the PDF text layer and falls back to OCR for scanned pages. We rank likely prescription sections, extract exercise claims with source evidence and confidence, detect conflicting values, and validate the result before generating gameplay.

Agent actions:

Clear note: Generates a personalized minigame and launches it.
Conflicting note: Blocks generation and asks a targeted clarification, such as: “I found both 8 and 10 repetitions. Which plan is newer?”
Missing detail: Asks for the missing left/right side or required equipment rather than guessing.
Camera or hardware disagreement: Pauses scoring, marks the attempt as unscorable, and requests recalibration.
Validated plan: Produces a constrained GameSpec that maps prescribed motion to game controls.

This creates entirely different games from different therapy needs:

Shoulder flexion can become a rock-climbing game where the arm trajectory controls reach
Grip exercises can become punching games where hand closure or wearable pressure dictates force
Wrist movement can become a navigation game where it guides an airship through gates

The agent composes a validated game specification from approved mechanics, movement mappings, difficulty bounds, and safety constraints.

Agent Hosting

Cloudflare hosts our persistent Therapy Game Agent.

Each therapy session gets durable agent state that stores the extracted plan, clarification status, generated GameSpec, calibration result, camera-confidence events, and completion summary. The browser receives live updates through WebSockets, so the game can immediately react when the agent pauses scoring, requests recalibration, or approves the next challenge.

A Cloudflare Workflow processes the completed session in the background: it saves summarized trajectory metrics, records low-confidence events, and prepares a therapist-readable summary without storing raw webcam video.

🧩 Challenges we ran into

The IMU axis was displacement, not velocity: We first detected slashes in our Zelda game from rate of change, which never fired for a slow, deliberate rehab sweep. Rewriting around displacement past a calibrated zero took the gesture from intermittent to reliable across every sweep size we tested.

Recalibration timing broke the gesture many times: Sending the re-zero command on the strike re-zeroed at the far end of the sweep, making the next slash unreachable. Sending it as the hand passed back through centre ate the margin small sweeps need. Only sending it once the hand has settled at rest works.

Rendering was GPU bound: The paintball city had shadow casting on every mesh, drawing the entire district twice per frame. We profiled with EXT_disjoint_timer_query_webgl2 and cut draw calls from 4,110 to 471, GPU frame time from 10.2ms to 3.4ms, and CPU from 5.0ms to 1.5ms.

🏆 Accomplishments that we're proud of

We had never built a game before this weekend, and were able to build three (even one in 3D using Three.js + WebGL) that were driven by live hardware input!!

Calibration is so hard! Nearly every bug that looked like a game bug was a sensor convention we had misread, and the fix was always to figure out the real signal/check the wires rather than tune a threshold. Since this is related to medical technology, we learned a lot about constraining AI models (cited evidence, schema validation, and clamped bounds) to make it safer to put near real people/patients!

🚀 What's next for RePlay

Improve Bluetooth reliability, extend forearm and grip to the upper and lower body, and give therapists a long-term overview dashboard of session history to create custom games that fit their improved progress.