MasteryMove

Dashboard of the App
Detecting skeletal postures using Media Pipe
Precision metrics calculations and live feedback
Coach Apex using Gemini Live
Precision metrics calculations and live feedback
Session Summary

Inspiration

When my grandmother underwent surgery, and a physical therapist came to her house every day, I realized: Why not create an app that lets anyone, anywhere, do exercise and receive physical therapy? That's how I was inspired to create this project.

What it does

MasteryMove uses a user's camera to map out landmarks on their body— on their face, arms, torso, and legs—and uses them to calculate the optimal angles to figure out the perfect form.

How we built it

I used Google AI Studio as the code editor and canvas for this project. I also used Google's MediaPipe SDK to map out many points on the user's body to allow for complex geometric calculations.

Challenges we ran into

Understanding Media pipe API Deploying Websocket for Google Gemini API

Accomplishments that we're proud of

Real-Time Performance Biomechanical Precision to generate 3D coordinates for skeletal Balancing Privacy with Cloud AI by running the model locally Models can run in phone browser as well

What we learned

How to leverage bidirectional WebSockets to stream low-latency live audio successfully bridging a React 18/Vite frontend with a proxy Node.js backend Gemini Multimodal Live API to create "Coach Apex" MediaPipe's Holistic model in the browser, extracting 3D skeletal, hand gesture, and facial coordinates.