Movement Coach AI

Inspiration

When exercising without a coach, it can be difficult to know whether your movement is consistent or whether your form could be improved. I wanted to explore whether computer vision and AI could provide useful movement feedback using only a camera.

I also wanted to make the idea privacy-conscious. Instead of sending raw exercise videos to a cloud service, Movement Coach AI processes the video in the browser to extract pose landmarks first.

What it does

Movement Coach AI is an AI-powered movement coaching web app that analyzes exercise videos and provides feedback.

Users can record or upload a video and select an exercise. The app uses MediaPipe to detect body pose landmarks and analyzes movement patterns over time.

It currently supports exercises including squats, lunges, pushups, bicep curls, dumbbell rows, shoulder presses, lateral raises, jumping jacks, situps, and tricep extensions.

The system can:

Detect body pose landmarks
Recognize exercises
Count repetitions
Measure movement characteristics such as joint angles and range of motion
Provide AI-generated coaching feedback
Generate personalized workout plans

How we built it

The frontend was built with React and Vite. It handles the user interface, camera input, video uploads, pose detection, results, and workout planning.

MediaPipe runs directly in the browser and extracts pose landmarks from the user's video. These landmarks are then processed into movement features such as joint angles and ranges of motion.

The backend was built with Python and FastAPI. It processes the movement data using NumPy, SciPy, Pandas, scikit-learn, and OpenCV.

A machine learning model is used as part of exercise classification, while the movement analysis pipeline looks at how body measurements change over time to detect repetitions.

For coaching feedback, the analyzed movement information is passed to an LLM through the Groq API. The AI then turns the movement measurements into more understandable coaching feedback.

The frontend and backend are deployed separately using Render.

Challenges we ran into

One of the biggest challenges was repetition detection. Detecting a person's pose in a single frame is relatively straightforward, but detecting a complete repetition required analyzing how the movement changes over time.

Real-world camera input also introduced problems that were not obvious when testing with controlled data. Different camera angles, lighting conditions, movement speeds, and partially visible bodies all affect pose detection.

There were also several technical issues during development, including Python dependency compatibility, MediaPipe setup, frontend-backend communication, CORS configuration, camera stream handling, and pose-processing timestamps.

Getting the application deployed was another challenge. The frontend and backend had to communicate correctly after being moved from a local development environment to separate Render services, and API keys had to be handled through environment variables.

Accomplishments that we're proud of

The final system connects browser-based pose detection, movement analysis, machine learning, repetition detection, AI coaching, and a workout planner into one application.

I'm especially proud that the project processes pose information in the browser instead of requiring users to upload their raw exercise videos to the backend.

I also managed to take the project from an initial idea to a deployed web application with support for multiple exercises.

What we learned

This project taught me how much more complicated movement analysis is compared to simply detecting a person in an image.

I learned how pose landmarks can be converted into useful movement features such as joint angles and ranges of motion, and how those features can be analyzed over time.

I also learned about building and connecting different parts of a full-stack AI application. React, MediaPipe, Python, FastAPI, machine learning, an LLM, and deployment all had to work together.

A major lesson was debugging. Many of the problems I encountered were not caused by a single large issue, but by small problems between different parts of the system.

What's next for Movement Coach AI

There are several areas I would improve with more development time.

I would make repetition detection and form analysis more robust across different camera angles, lighting conditions, and movement styles.

I would also expand the exercise library and improve the machine learning models with more varied training data.

Another major feature would be progress tracking, allowing users to compare their movement and performance across multiple workouts.

I would also continue improving the privacy, accessibility, and safety of the system while making the coaching feedback more useful without presenting it as medical advice.