RoastCoach

A computer vision–based system for real-time fitness form correction.

Comment

Inspiration

Over winter break, a close friend of the RoastCoach team tore her ACL while skiing and returned to college facing a major obstacle in her recovery: she had no reliable way to get to physical therapy. Without consistent access to a physical therapist, she was left to perform her rehab exercises alone, unsure whether she was doing them correctly. Small form mistakes added up, slowing her progress and ultimately led to her extended recovery lasting a little over a year! Watching her struggle made it clear that recovery and proper movement guidance shouldn't depend on proximity to care or transportation. RoastCoach was inspired by her story and wanted to create something accessible to everyone, whether its guiding physical therapy exercises, tracking fitness goals, or preventing injury.

What it does

RoastCoach analyzes exercise videos to provide personalized, form specific coaching. It extract body joint markers from video, computes joint angles and movement metrics, segments repetitions, and compares a user's motion against a reference "coach" movement sample. Based on validated biomechanical deviations, RoastCoach generates short, targeted feedback cues. All workout videos and analyses are stored and can be revisited in the History tab with a calendar view.

How we built it

We built RoastCoach using an industry standard biomechanics pipeline. Video input is processed with a pose estimation model to extract body joints, which are smoothed to reduce noise. Joint angles and derived metrics such as squat depth, and symmetry are computed using deterministic vector math. A driver joint signal is used to segment repetitions, which are time normalized and compared against a reference motion sample built from a coach video. Rule based validation detects form issues, and a constrained LLM layer converts these validated issues into human readable coaching cues. We used Streamlit for the interface and AWS S3 to store uploaded videos, analysis artifacts, and session history.

Challenges we ran into

One of the biggest technical challenges we faced was reliable repetition detection. Rep detection is important because all downstream analysis depends on cleanly segmented reps, which are needed to normalize motion over time and compute meaningful averages. Without accurate rep boundaries, comparisons between a user's movement and a reference become unstable or misleading. Pose estimation is inherently noisy, and when working with the data, we noticed the noise caused valid reps to be split, merged, or discarded, ruining the average motion profile. We addressed this by smoothing joint trajectories and applying temporal constraints to filter out random motion.

Accomplishments that we're proud of

We're proud of designing a minimalistic UI to help people track their progress towards performing exercises with stronger form. This project would help personal goals of improving calisthenic and workout form, as well as provide accessible feedback to those in need of it.

What we learned

During this project, we learned how to utilize Amazon buckets to store and fetch information onto a UI, as well as building quickly. Additionally, through working closely with the data pulled from the sample videos, we learned about the importance of data validation and system reliability.

What's next for RoastCoach

The next step would be to deploy an app that includes minor enhancements to improve user experience. This could include real time feedback so as the user is performing the exercise, RoastCoach could inform them of improvements live through a text to speech API.

Built With

Share this project:

Updates