-
-
Hardware setup. Breadboard system is powered by a powerbank, lone ESP is connected to the laptop.
Inspiration
Training alone makes it hard to know if each rep is structurally sound with proper form. It’s easy to slack on depth or symmetry without noticing, so we aimed to build a system for live feedback, both digital and physical. We wanted a quick and simple way to verify whether we were performing bodyweight exercises with good form, no personal trainers or workout partners needed.
What it does
Formify estimates poses from the webcam, tracks push-ups, squats, and sit-ups, and counts reps while scoring whether each rep meets form constraints. A browser UI shows the live video stream with skeleton point overlay, and metrics over a WebSocket. A hardware component drives a buzzer on bad form, giving live audio output so feedback is hard to ignore during a set.
How we built it
Software: A FastAPI backend grabs frames with OpenCV, runs MediaPipe Pose Landmarker, and routes landmarks into small trackers for each exercise. Each tracker computes joint angles and applies smoothing so validity is determined using stable geometry. We also implemented movement states (e.g. up/down) to act as baselines. Trackers expose when and where integrity drops used for both UI highlighting and hardware alerts. The frontend is React + TailwindCSS, and a WebSocket carries data so the browser remains light.
Hardware: The system runs off two ESP32 boards. One ESP32 plugs into the laptop through USB in order to export error data. That ESP32 talks peer-to-peer to a breadboard ESP over ESP-NOW so the remote board can drive the buzzer without the laptop joining another wireless network. This eliminates the need for connection to WiFi or Bluetooth for ESP connection. Additionally, a rate limit on alerts keeps the buzzer from firing on single-frame noise.
Challenges we ran into
Software: Threshold tuning for tracker angles was slow and tedious, since we had to experiment with varying degrees of strictness in relation to “correct” landmarks. Getting webcam data also varies depending on platform, so we had to work through a setup that cycles through camera indices to find one that works for the device running the app.
Hardware: We drastically changed our hardware plans fairly late into the hackathon, leading to major changes in the Python scripts that supported our ESPs. We originally planned for a haptic handmount system that would vibrate when form deteriorated. However, our vibration motors weren’t delivered on time, leading to us having to deviate from this plan. We also had issues with the buzzer activating on brief, single-frame issues. To prevent this, we added rate limits and delays.
Accomplishments that we're proud of
We’re proud of having one coherent pipeline: poses ran through geometric rules, data displayed on the Web UI, and physical alerts driven by the same logic validity logic. We’re also proud of the decision to use ESP-NOW, which keeps the remote buzzer off Wi-Fi and Bluetooth, allowing for simple plug and play action.
What we learned
Geometry using MediaPipe and OpenCV wasn’t as simple as we predicted. There was extensive tuning and angle calculations that had to be continually edited and recalibrated based on exercise. We also learned how to use ESP-NOW as an alternative to traditional wireless microcontroller connections. Additionally, we learned to create backup plans before building, as it would have saved us a lot of time if we had ideated our current buzzer system earlier instead of relying on the initial plan to work as intended.
What's next for Formify
There’s a slew of ideas for features we had but ran out of time to implement. Firstly, we could expand to more movements beyond the three we have right now. We could also add session history, with a dashboard to view integrity over time and how it fluctuates. Also, given the time to acquire some more hardware products, we could design and implement our original idea of a haptic handmount to provide vibration feedback when integrity drops.
Log in or sign up for Devpost to join the conversation.