FormFusion

Revolutionize your workouts with AI! Get instant form feedback. Compare to a pro's movements. Track and improve joint angles with annotated video. Elevate your fitness to the next level.

Comment

Inspiration

When we first arrived at the hackathon, we were unsure what project to pursue. But through an intensive ideation process, we found common ground in our shared passion for fitness and technology. The result was the creation of an AI-powered app that provides instant feedback on your form during physical activity. It's the perfect combination of our interests and expertise, and we're thrilled to bring this innovative solution to the fitness community.

What it does

Our web app leverages the power of AI to analyze and improve your workout routine. Users upload recorded exercise videos, which are processed by the proprietary machine learning API that we wrote. This technology provides instant feedback on form, enabling users to optimize their workouts. The annotated videos, including information about joint angles, are then displayed within the app for easy review and tracking of progress.

How we built it

We built the app using a Flask framework for the backend and HTML/CSS for the frontend. The user-friendly interface allows for the upload of two videos and any desired configurations. The videos are then sent to an AWS S3 bucket and processed by our custom OpenCV and Mediapipe API, run locally and tunneled via ngrok. This program analyzes the user's movements and provides real-time feedback on their form. The annotated videos, along with written feedback, are then uploaded back to the Flask app and displayed to the user through the HTML/CSS interface.

Challenges we ran into

Building this app was a journey filled with challenges and obstacles, but our team persevered through hard work and a willingness to learn. One of the biggest challenges was finding an effective way to run the machine learning function. Initially, we attempted to use AWS Lambda with layers for OpenCV modules, but we ran into issues with the large size of dependencies. We eventually shifted to a HTTP API, which proved to be a more viable solution. On the frontend, our team member who was new to HTML/CSS faced some hurdles but rose to the challenge and learned through documentation, resolving bugs along the way. Currently, we are facing a challenge with the Flask backend, but we are actively working to troubleshoot and find a solution. Despite these obstacles, our team is dedicated to delivering a top-notch product and pushing through any difficulties that arise.

Accomplishments that we're proud of

We have accomplished several key milestones in the development of this app that we are extremely proud of. Our custom machine learning script accurately calculates the angles for body joints during motion, providing users with detailed and accurate feedback on their form. We have also integrated cloud storage architecture into our app, allowing for efficient and seamless processing of videos. In addition, we have split the app into different services to reduce the load on the client side, ensuring a smooth and seamless user experience.

What we learned

Working on this project has been a learning experience for the entire team. With each person taking on a specific role, we have had the opportunity to expand our skills and knowledge. Two team members were responsible for the frontend, with one of them learning HTML/CSS from scratch. Another team member tackled the Flask app, integrating it with AWS boto3 for the first time. The final member of the team was responsible for learning the Mediapipe API and creating the ML server on ngrok. Through this process, we have all learned new technologies and gained valuable experience that will serve us well in future projects. The opportunity to work together and grow as a team has been one of the highlights of this project.

What's next for FormFusion

With the successful completion of our hackathon project, we are now looking to the future. Our next goal is to build a library of pre-recorded workouts and activities, which we will host on an EC2 instance to ensure 24/7 accessibility. This addition will allow users to compare their form against a wider range of professionals and activities, taking their feedback to the next level. With this new feature, we hope to continue improving the user experience and further solidifying our position as a valuable tool for anyone looking to enhance their physical activity.

Built With

Share this project:

Updates