Our idea is to make a personal trainer for everyone that works in the comfort of your own home. Not only can you take this trainer anywhere, but you can also get dynamic feedback from it.
What it does
Our app provides real-time feedback about your motions by using advanced image processing to analyze your exercise. We use machine learning to compare your motions with ideal motions and tell you how to improve your technique.
How we built it
In the front end we have a webcam running alongside* with Alexa. Once prompted by an Alexa intent call, the local computer records you performing your exercise and sends it to an Azure back-end server. The Azure backend then shoves the video frames into way too many convolution layers that gives a sequence of points that indicate your joints. It then analyzes this sequence of points and studies the dynamics of your joints. The result is then returned as verbal feedback, ex: "Your stance should be wider."
Challenges we ran into
We initially tried to run the model locally. However, without a GPU this proved to be too slow. We then migrated our server code to an Azure VM with a GPU. We were unable to figure out how to make our model perform batch processing, which makes the analysis slower than expected.
We also had trouble establishing the communication between the Lambda functions for Alexa hosted on AWS and the Azure server/our local computer. As a result, we didn't have the vocal prompts and automation at the moment.
Accomplishments that we're proud of
The squat analysis via webcam works relatively well and gives pretty fast and accurate feedback on your squat (dependent on the internet). We were able to tune our model to detect the problems with a given squat and relay this verbally.
What we learned
Vision is hard.
What's next for MoreConvidence
Optimize the model to be able to run in real time, locally, on a mobile phone, and incorporate voice activation. Higher quality images would lead to a better description of joint locations. This would allow us to analyze a greater range of exercises.