Inspiration

Most motion-controlled games assume every player can stand, jump, or use a traditional controller. For children who use wheelchairs or have limited lower-body mobility, that can mean watching their friends play from the sidelines.

We built umove because no child should experience that kind of FOMO.

Our goal was to create a game that is accessible from the beginning. We also wanted movement to feel exciting rather than clinical. That is why we incorporated upper-body stretches, controlled reaches, side leans, posture work, and coordination exercises commonly recommended by movement and rehabilitation experts into playful missions.

What It Does

umove turns an ordinary webcam into a hands-free motion controller.

Players can choose from several activities:

Stretch & Flow: Complete guided movements to find parts, restore power, and repair Moshi’s spaceship.
Flying Adventure: Flap to gain altitude, lean to steer, and fly through glowing rings.
Boxing Beat: Punch, block, duck, and dodge a robot opponent while earning points and building combos.
Robot Mirror Buddy: Watch Moshi copy the player’s movements in real time.

Players can also control the interface without touching a mouse or keyboard. They can:

Move either hand, or both, to control on-screen cursors
Hover over buttons to select them
Swipe between games
Cross their arms to return to the previous screen

Accessibility options include adjustable dwell timing, larger text, high contrast, color-vision filters, reduced motion, sound cues, and support for different selecting hands.

How We Built It

umove runs directly in the browser using JavaScript, MediaPipe, Three.js, WebGL, and Vite.

MediaPipe detects 33 body landmarks from the player’s webcam in real time. We normalize those landmarks using measurements such as shoulder width and torso height. This allows the controls to respond to the player’s proportions and distance from the camera.

Our motion pipeline converts raw landmarks into reusable actions such as:

Reach
Lean
Flap
Punch
Block
Select
Go back

We created an accessibility-mapping layer that can translate lower-body actions into arm or torso movements. This allows games to share the same actions without assuming that every player moves in the same way.

We also built a custom 3D robot named Moshi. A Three.js retargeting system maps the player’s pose onto Moshi’s skeleton using world-space bone rotations, motion smoothing, visibility checks, and confidence fallbacks.

Each game uses specialized motion analysis. Boxing punches use world-space depth, while Flying Adventure detects flaps using arm position and movement velocity.

All camera processing happens locally inside the browser. Frames and body landmarks are never uploaded or recorded.

Challenges We Faced

One of our biggest challenges was making motion tracking responsive without making it overly sensitive. Small tracking errors could accidentally trigger an action, while strict thresholds could make the game frustrating for players with a limited range of motion.

We addressed this with:

Body-relative normalization
Confidence thresholds
Gesture cooldowns
Pose calibration
Motion smoothing

Retargeting human movement onto a stylized robot was another major challenge. Human joints and Moshi’s bones have different orientations and proportions. We preserved the robot’s original bind pose, calculated rotations in world space, and converted them back into each bone’s local coordinate system.

We also had to design gameplay and navigation together. Creating accessible games would not matter if players still needed a mouse to launch them. That led us to build hand cursors, dwell selection, swipe navigation, and gesture-based back controls.

Accomplishments We’re Proud Of

We are proud that umove is more than a technical pose-tracking demo. It is a complete, playable experience featuring:

Multiple games
Guided physical activity
Scoring and combo systems
Interactive missions
Sound feedback
A custom animated character
Hands-free navigation
Meaningful accessibility settings

Most importantly, accessibility is part of the core architecture instead of an afterthought.

What We Learned

We learned that accessible design does not have to reduce challenge or fun. The same objective can remain intact while the movement used to complete it changes.

We also learned that raw pose detection is only the beginning. Building a good motion-controlled experience requires calibration, normalization, clear feedback, forgiving thresholds, and careful handling of tracking loss.

Designing for different bodies ultimately produced a more flexible interaction system for everyone.