Inspiration
Physiotherapy and rehab work depends on repetition, and repetition is boring patients disengage, skip sessions, and recovery stalls. We wanted to turn the exercises a physiotherapist actually prescribes into game input, so the reps that rebuild muscle control also move you through a world you want to keep playing.
What it does
ReWorld is a browser game you play with your body. A camera reads your physiotherapy movements — [opening palms, raising an arm, rotating your neck] and translates them into control of a fox exploring a photoreal 3D world. The exercises your therapist recommends are the controller, so every rep is progress in the game and progress in recovery.
How we built it
Motion input: OpenCV reads the webcam feed and detects the player's physiotherapy movements, mapping them to the fox's actions in real time. World: fal (nano-banana) for the seed image → World Labs Marble for the Gaussian-splat environment → rendered in-browser with Spark and react-three-fiber. Character: the fox, built in Tripo, moving through the space. Stack: React + Vite, runs in any browser, deploys to [Vercel].
Challenges we ran into
Mapping messy human motion to game input. Real physiotherapy movements are slow and vary between people; getting OpenCV to turn them into reliable, forgiving controls without frustrating someone mid-recovery took real calibration and tuning. Splats have no geometry. The photoreal world is just colored blobs with no floor or walls, so we built the fox's collision and movement bounds by hand against the visible space. Reconciling coordinate systems between the splat and the character mesh, so the fox stands and moves correctly in a world that has no real surfaces. Accessibility over responsiveness. The whole point is that someone with limited mobility can play, so controls had to work without demanding speed or precision.
Accomplishments that we're proud of
A game genuinely controlled by therapeutic movement webcam and OpenCV, no special hardware. A photoreal generated world that makes rehab feel like exploring somewhere, not doing drills. A full pipeline chaining generative world (World Labs), character (Tripo), seed imagery (fal), and real-time motion input (OpenCV) into one accessible experience. Shipped in 18 hours, entirely in the browser, no game engine.
What we learned
Designing for recovering patients inverts normal game design: forgiving beats responsive, encouragement beats challenge. Computer-vision input has to tolerate the variability of a real human body, not an idealized one. Splats are set dressing, not spatial data build the interactive layer separately.
What's next for ReWorld
Configurable exercises so a physiotherapist can map specific prescribed movements to controls per patient. Progress tracking range of motion and reps over time, shared between patient and therapist. More worlds and fox behaviors to sustain engagement across a long recovery. Validation with real physiotherapists and patients.
Log in or sign up for Devpost to join the conversation.