Diseases such as heart disease and diabetes are among the leading causes of death in the United States. Our lack of concern for our own health is causing premature death yet less than 20% of Americans get enough exercise. The pandemic only escalated those health problems too. A study published in the journey Obesity recently discovered that the average individual engaged in sedentary leisure activities 21 minutes more per day during the COVID-19 pandemic - most of which is spent online engaging in technology.

Admittedly, the world is becoming more and more reliant on technology - we're moving towards a digital future. But advancements in technology shouldn't equate to a decline in our fitness. We believe that technology and health can, and should, go hand in hand. Innovations such as machine learning and real-time communication could not only support healthy living, but revolutionize it. These technologies could help people connect with personal coaches, help people correct their postures, and help people keep track of their workout routines.

We want to help millions of more people live healthier lifestyles so we did all that, and more, with - a universal fitness application for achieving the perfect workout.


What is is a revolutionary web application that utilizes cutting-edge technology to help users experience the perfect workout routine. We designed it from scratch to ensure that all users are able to enjoy the feeling of exercising. We then built it using real-time web technologies (WebRTC & WebSockets) and complicated data analytics (Machine Learning & Mathematical Formulas) to create the best platform for working out. Here's how it works.

1. helps you achieve the perfect workout! uses advanced algorithms and modern web technologies to help you create the perfect workout routine.

Splash Screen

2. Workout anytime, anywhere! allows you to schedule live sessions with professional instructors or play back prerecorded workout routines for your convenience.

Schedule Screen

3. Automatically track your workouts! uses machine learning and data to automatically detect your exercises, calculate calories, and determine active time.

Exercise Detection

4. Compare with a trainer! compares your posture and performance with those of your instructors' to ensure that you're doing your exercises perfectly.

Compare Screen

Design was designed using the double diamond design process, a model popularized by the British Design Council.

Double Diamond Design Process We followed the four stages of a double diamond process - Discover, Define, Develop, Deliver.

In order to really focus on our users, we completed as many steps of a design process as we could over the course of a weekend. We conducted need-finding, created user personas, created a design system, and tested low fidelity and high fidelity prototypes. For more information, view the sub-sections below.

User Personas

User Personas We created primary and secondary user personas to help understand our problem.

Low Fidelity Prototypes

Lo-Fi Prototypes We created wireframes and low-fidelity prototypes to help us rapidly design the perfect UI.

Design Language

Design Language We created mood boards, color scenes, and font choices to help us prototype.

High Fidelity Prototypes

Figma We combined our prototypes, our design language, and our user testing to create a final prototype.


System Architecture

System Architecture Diagram We carefully designed our system for speed and scalability.


Our backend is primarily built on a tech stack of Node.js and deployed on Heroku. The two create a WebSocket server that transmits body pose keypoints allowing for real-time pose comparisons between the instructor and the user. These comparisons were made using Cosine Similarity, a model which allows us to measure similarity between two non-zero vectors, where each vector consists of the positions of the person's body parts. After normalizing the vectors and calculating the cosine similarity, we would find the Euclidean Distance which takes into account a user's relative positioning in the rating calculations. These calculations are made in real-time to output a final confidence score, between 1 and 100, that is used to rate the user's workout in comparison to the instructor. To our knowledge, we are the first people to implement real-time pose similarity between multiple video feeds.

Cosine Similarity We used cosine similarity in order to calculate the similarity between different poses.

Euclidean Distance of Cosine Similarity We used euclidean distance of cosine similarity to compare the poses of instructors and students.


Our frontend is a web application built primarily on a stack of React, Javascript, SASS, Ant Design and deployed on Netlify. Our web application utilizes Tensorflow, specifically the PosetNet model, and the user's webcam camera, to handle pose estimation and exercise detection. The PoseNet model gives us the keypoints of 17 different joints including each joint's position and confidence score. We leveraged the angles between different joints using these keypoints in order to detect when a user is performing a particular exercise (for example, a squat, jumping jack, or bicep curl). By using angles, we were able to create a scalable system of exercise detection that can be sent to our backend in order to perform similarity comparison.

Another important core technology we used was WebRTC to allow for real time sessions with instructors. We leveraged Google Cloud Firestore as a signaling server in order to generate new WebRTC offers and SDPs. We also used Google's free STUN servers for communication. In this way, our application is scalable to allow for a vast amount of concurrent calls.

Technologies All of the technologies that went into


Blackwell DL, Clarke TC. State Variation in Meeting the 2008 Federal Guidelines for Both Aerobic and Muscle-strengthening Activities Through Leisure-time Physical Activity Among Adults Aged 18-64: United States, 2010-2015. Natl Health Stat Report. 2018 Jun;(112):1-22. PMID: 30248007.

Ducharme, J. (2019, June 14). Can Americans Hack Their Way Out of an Aversion to Exercise? Time.

Flanagan et al., 2020 E.W. Flanagan, R.A. Beyl, S.N. Fearnbach, A.D. Altazan, C.K. Martin, L.M. Redman. The impact of covid-19 stay-at-home orders on health behaviors in adults. Obesity (2020). Google Scholar

Move Mirror: An AI Experiment with Pose Estimation in the Browser using TensorFlow.js. (2018). Tensorflow.Org.

Phillips, T. (2019, August 7). Exercise Classification with Machine Learning (Part I). Medium.

Thomas, L. (2020, November 6). Peloton says recent spike in Covid-19 cases, lockdowns are boosting sales. CNBC.


What We Learned

In order for the system to work, there were a lot of features that needed to be implemented. First, we needed a system to share a live video and audio feed between the instructor and the client. To accomplish this, we followed a WebRTC tutorial to be able to establish a peer-to-peer connection. Next, we needed to analyze the video feed to determine their current post. This could be achieved using Tensorflow, which is open source platform for machine learning. We learned about how to leverage that platform to analyze the video feed and return an array of coordinates corresponding to the current positions of a user's body, including the hands, elbows, shoulders, legs, and feet. Using these data points, we needed to determine if a specific pose was being executed. We learned about built-in math functions that Javascript offered, in order to match up position vectors to a pre-set condition for each position. Finally, we needed to compare two unique poses and return a metric to indicate how similar they were. To accomplish this, we used cosine similarity to return a single numeric value determining the similarity of the poses. After all these steps complete, we provide feedback to the user regarding the data that was just analyzed. Over the past weekend, we learned a lot about new web technologies and libraries that we could incorporate into our project to meet our unique needs.


There were a lot of technologies that we needed to integrate to create a working project. We used WebRTC to establish a live video and audio connection between the instructor and the client. Using this feed, we were then able to use Tensorflow to analyze the live video to estimate the pose of a person. After obtaining the video feed, we detected a specific exercise motion, like squatting or curling. We were then able to transmit the detected poses back to the backend to compare the similarities using cosine similarity to generate an accuracy score. Over the past weekend, we got all of these steps to a working level to create a functioning project.

What's Next

A big challenge of the project was being able to detect the specific poses and match them between the instructor and the client. Given more time, we'd be able to determine the movement patterns that come from a variety of poses and recognize them. Also, we'd like to add a dance mode that would generate live pose detection to compare the instructor to the client based on general positions rather than a specific move.

+ 29 more
Share this project: