Gym Vision

Where things workout.

Comment

Inspiration

We were chilling in a gym and wanted to build the perfect summer bodies, but we didn't really know how, not to mention some of these equipment straight up looked like it was from outer space. This sparked the perfect idea, to build a website that's easy to use, plans everything out for you, and on top of that, has customizable voices!

What it does

Gym Vision can build you an entire gym routine based off of what you have available and what you want to work on. If the equipment isn't relevant, then Gym Vision recommends some bodyweight exercises like pushups and more. One key feature is that if you don't know how to do what it recommends you, it has a link that sends you to YouTube to learn it! You can also choose the intensity of the workout and how your training coach should sound like.

How we built it

We built Gym Vision as a full-stack web app with a lightweight React frontend and serverless backend, layering in multiple free AI services for equipment detection, voice, and motion tracking rather than relying on any single model.

Frontend

  • React + TypeScript + Vite
  • Zustand for state management
  • Custom CSS design system (glassmorphism, warm color palette)
  • TensorFlow.js + MoveNet (pose detection) — runs entirely client-side for live rep counting and hold timers

Backend

  • Vercel serverless functions (Node.js + TypeScript)
  • Deployed and hosted on Vercel

AI / Equipment Detection

  • Roboflow (custom-trained YOLO-style computer vision model) — primary equipment detector
  • Google Gemini Vision API — second detector run in parallel as an ensemble, catching equipment the first model misses
  • Custom equipment-alias mapping layer to normalize both models' outputs into one shared vocabulary

Voice Mentor

  • ElevenLabs Text-to-Speech API — multiple distinct character voices with contextual motivational scripts

Workout Logic

  • Custom-built exercise-matching engine: maps detected equipment to a curated exercise database, with automatic bodyweight substitutions when equipment is missing
  • YouTube integration for exercise demo links

Infrastructure

  • Vercel (hosting + serverless functions + environment/secrets management)
  • Free tiers only across every service (Roboflow, Gemini, ElevenLabs, Vercel)

Challenges we ran into

At first, the Program couldn't even detect a dumbbell if it was right in front of it! 9 out of 10 times, it failed. After many unsuccessful attempts, we had the idea to use more than one model for recognizing the objects. With the help of Gemini, we finally had a working product! After a lot of testing, it remained flawless, except for the issue with limited requests per day due to the free version of Gemini. If we actually launch this, we will make sure to use premium servers for optimal customer satisfaction.

Accomplishments that we're proud of

We're proud of the fact that our program could successfully recognize a wide variety of equipment. We were also super proud when our website could track reps in real time!

What we learned

We learned how to optimize our selection of AI to reach an acceptable end result. We also learned that recognizing an object through the eyes of a computer is harder than it looks!

What's next for Gym Vision

We hope to be able to roll out a wider variety of personalities and tones for our training coaches to make training feel like a day with your best friend.

Built With

Share this project:

Updates

Submission history