FormFriend

Inspiration

As a team of STEM interns, we spend a lot of time sitting at our desks and understand how crucial it is to stay physically active. We were all interested in fitness for its incredible health benefits, but we quickly ran into the common barriers: the high cost of personal trainers, the risk of injury from improper form, and the feeling of self-consciousness that can come with going to a gym.

We asked ourselves: Could we use AI to democratize personal fitness, making expert guidance accessible and affordable? That question was the inspiration for FormFriend. We wanted to build a tool that could give anyone the confidence to start their fitness journey safely and effectively, right from the comfort of their own home.

What it does

FormFriend acts as a comprehensive, AI-powered personal trainer that guides users from initial setup to a complete, form-corrected workout. It seamlessly integrates three core features to create a supportive and effective fitness experience.

1. Personalized Workout Planning The user journey begins with a simple, intuitive onboarding process where they input their personal stats (like age and weight) and fitness goals (such as 'build strength' or 'weight loss'). Based on this data, our backend service leverages the Gemini API to generate a dynamic, tailored workout plan, providing a clear roadmap for the user's fitness journey and eliminating the guesswork of what exercises to do.

2. Interactive AI Coaching Once a plan is created, our conversational AI coach, "Jackie," greets the user with a personalized audio introduction using the Vapi API. This creates an engaging and supportive experience from the start. The conversational interface allows users to ask questions and receive on-demand guidance, making the experience feel closer to working with a real trainer.

3. Real-Time Form Correction This is the core feature of FormFriend. When a user starts their workout, the app uses their webcam and the TensorFlow.js MoveNet model to analyze their exercise form in real-time. It visually maps a skeleton onto the user's body, calculates the angles of their joints, and provides immediate, on-screen feedback to help them correct their posture and technique. This significantly reduces the risk of injury and ensures every repetition is effective, building a strong and safe foundation for their fitness.

How we built it

FormFriend is built on a modern web stack designed for a rich, interactive user experience.

1. The User Foundation: We chose a robust frontend using React and TypeScript, powered by the Vite build tool for a fast and efficient development workflow. The user interface was crafted with a combination of shadcn/ui and styled-components for a clean, polished, and responsive design.

2. The AI Form Coach: The core of FormFriend is the real-time form analysis. We integrated TensorFlow.js with the lightweight MoveNet pose detection model. This allows the application to track 17 key body joints through the user's webcam. Our custom algorithms then calculate the angles between these joints in real-time to provide immediate, actionable feedback on exercises like squats and lunges.

3. The Conversational Trainer: To make the experience more personal and engaging, we implemented a conversational AI coach using Vapi. This voice agent, "Jackie," provides a personalized audio introduction based on the user's goals and is the foundation for future real-time, voice-based interactions. Personalized Plans: The workout plans are generated by a custom backend service built with Node.js and hosted on Render, which takes the user's stats and goals to create a tailored week-by-week fitness schedule.

Challenges we ran into

A main challenge we faced was ensuring the TensorFlow pose detection ran smoothly without lagging the user's browser. We had to be mindful of our state updates and the frequency of our detection loop to balance accuracy with performance, creating a seamless user experience.

Accomplishments that we're proud of

Mastering Frontend Development from Scratch: As a team with no prior frontend experience, we are proud to have built a complete, responsive, and modern web application using React and TypeScript. We learned and applied concepts like component-based architecture, state management with React Hooks, and asynchronous data fetching.

Integrating Complex AI in the Browser: We successfully implemented two advanced AI features. The core of our project is the real-time pose detection using TensorFlow.js, which required us to process a live video feed and perform calculations directly on the user's device.

Implementing a Conversational Voice Agent: We went beyond a visual-only interface by integrating the Vapi API to create "Jackie," our AI audio coach. This taught us how to work with real-time, event-driven APIs and manage complex states like an active call.

Resilient Problem-Solving: The journey was filled with challenges, particularly in debugging the APIs and ensuring all services worked together seamlessly. We are proud of the persistence and systematic approach we developed to diagnose and solve these complex issues, which has made us better developers.

What we learned

This project was a deep dive into the practical application of machine learning in the browser. Implementing TensorFlow.js for real-time video processing taught us a great deal about performance optimization and managing complex state in a React application.

Integrating a sophisticated voice agent like Vapi was another major learning experience. We learned how to handle event-driven communication, manage sensitive API keys securely using environment variables (.env.local), and debug complex, asynchronous API errors. This process reinforced the importance of meticulous documentation, reading, and systematically isolating variables to find a root cause. Beyond the code, we learned about user-centric design, thinking through the entire user journey from their initial hesitation about fitness to feeling empowered and confident in their workout.

What's next for FormFriend

FormFriend is a powerful foundation, and we're excited about the many ways we can expand its capabilities to create an even more comprehensive and immersive fitness experience. Our roadmap is focused on making the AI coach smarter, more interactive, and more personalized.

Here are the key features we plan to build next:

Fully Interactive Real-Time Coaching: Our top priority is to evolve our AI coach, "Jackie," from an introduction-only agent to a fully interactive workout companion. By feeding the real-time form feedback from TensorFlow.js into our Vapi voice agent, Jackie will be able to provide spoken encouragement and corrections throughout the entire workout, with phrases like, "Great depth on that squat!" or "Two more reps, you've got this!"

Expanded Exercise Library: We plan to significantly increase the breadth of supported exercises. This includes adding more complex weightlifting movements (like cleans and overhead presses), popular bodyweight exercises, yoga poses, and even physical therapy movements to make FormFriend a versatile tool for a wider range of users and fitness goals.

Automatic Rep and Set Counting: By analyzing the angle data from the user's movements over time, we will implement automatic rep and set counting. The app will not only know if your form is correct, but it will also track your progress through the workout without requiring any manual input from the user.

Historical Progress Tracking: To help users see their improvements, we plan to add user accounts and a database to store workout history. Users will be able to look back at their past sessions, see charts of their strength gains over time, and stay motivated by tracking their long-term achievements.