Natural Motion

Inspiration

As college students and developers, we spend hours sitting at desks, which leads to back pain, muscle soreness, and stiffness. Stretching and physical therapy help, but it is hard to know which movements to do or whether you are doing them right, and seeing a physical therapist is not always convenient or affordable.

We wanted to make guided movement more accessible by combining AI, computer vision, and interactive 3D anatomy into a personalized, hands-free experience.

What it does

Natural Motion is an AI movement coach that helps users stretch and improve their mobility through natural conversation and real-time feedback.

Personalized recommendations: Users can describe how they feel physically, such as having tight shoulders or back stiffness, and the coach recommends and demonstrates supported movements tailored to their needs.
Interactive 3D demonstrations: An interactive 3D anatomical model demonstrates each movement while highlighting the target muscle groups, helping users see both how to perform a stretch and which muscles it works.
Live form tracking: When users follow along, Natural Motion uses their device's camera and computer vision to track body positioning and spot observable form issues, giving verbal guidance to help them adjust.
Hands-free voice control: Users can ask questions, request another demonstration, or change the pace without interrupting their session.

How we built it

We built Natural Motion as an Android-first mobile app using React Native, Expo, and TypeScript, connecting live conversation, animated movement demonstrations, and camera-based pose feedback.

3D Visual Coach: Built with Three.js and React Three Fiber to animate a rigged 3D anatomy model. We implemented composable shoulder and elbow movements with controlled transitions and holds, alongside educational highlighting of modeled muscle regions.
AI & Planning Pipeline: Gemini Live handles spoken conversation and sampled camera images through a WebSocket backend, while an OpenAI-powered planner translates movement requests into structured animation sequences. We added validation across the backend, app, and renderer to check supported movements, joint limits, and playback constraints before execution.
Pose Feedback: On-device landmark detection feeds a deterministic geometric check of elbow height and spread during a stationary goalpost hold. We distinguish an unclear camera view from meeting those criteria so feedback stays tied to what the camera can actually observe.

Challenges we ran into

Learning mobile development from scratch: None of us had prior mobile development experience, so learning a new ecosystem was tough even with a React background.
Upside-down camera frames: The pose model failed at first because the camera feed reached MediaPipe upside down due to the library negating the rotation angle. This showed up in the debug numbers: ears below the shoulders and the skeleton locking onto a face. We patched the Kotlin frame converter to rotate the pixels upright before inference and pinned it to the library version.
Custom native builds: Because Expo's camera does not expose frames, Expo Go could not run the pose module. We had to create a custom Android build, meaning every native change required a rebuild and reinstall.
Rigging and muscle highlighting: We needed a single 3D model that could demonstrate a movement and highlight the muscles involved without breaking the rig, which required careful rigging, animation, and muscle-highlighting work.
Safe agent control: We gave the agent tool calls that request supported movements instead of raw joint values, and checked every sequence against what the rig can actually do. The agent proposes; the app decides.
Cutting voice latency: Three models shared one loop: on-device pose detection, a live speech model for listening and speaking, and a movement planner. The first spoken reply took about six seconds, making the conversation feel slow. We made the movement tool non-blocking so the model could acknowledge the user while planning, discarded late results, and separated wall time from the phone's clock—dropping first-audio latency to under one second.
System integration: Fitting together the separate workflows we had built in parallel and debugging all the edge cases took a lot of late-night testing.

Accomplishments that we're proud of

Getting MediaPipe pose tracking working after hours of debugging.
Seeing the app run natively on a phone despite having no prior Expo or React Native experience.
Getting the voice model to converse almost instantaneously (under 1 second first-audio latency).
Watching all the separate pieces—vision, 3D rendering, and voice AI—finally come together as one working app.

What we learned

How to combine computer vision, interactive 3D modeling, and agentic AI into one working app.
How real-time pose estimation can work alongside anatomical animations to evaluate movement, and how an AI agent can translate natural-language requests into structured, validated instructions for a 3D model.
How to design a hands-free, conversational interface that ties verbal guidance to visual demonstrations and live camera feedback.

What's next for Natural Motion

Our long-term goal is to expand Natural Motion to cover full-body mobility:

Add support for lower-body stretches and more muscle groups.
Expand our library of supported movements and improve the accuracy of real-time form feedback.
Make the AI coach adapt better to individual mobility levels and preferences.
Build personalized stretching routines and track progress over time to make guided movement accessible to anyone, anywhere.