RepWise

RepWise is a live AI workout coach that builds your session, tracks your reps and form, and rescues your workout when gym friction gets in the way.

Comment

RepWise

Inspiration

Most people do not stop working out because they cannot count reps. They stop because training alone has too much friction.

Sometimes you do not know what to do that day. Sometimes the gym is crowded. Sometimes a machine is occupied, your energy is low, or you are not confident about an exercise. Beginners especially face uncertainty, and even intermediate users constantly adapt sessions on the fly.

RepWise was inspired by that gap. We wanted to build an AI system that does more than watch movement. We wanted to build a live workout companion that helps users start the right workout, stay on track during the session, and recover when the workout breaks down.

What it does

RepWise is a live AI workout coach that creates a session based on the user’s:

  • goal
  • experience level
  • available time
  • equipment
  • energy level
  • workout context

It then guides the session in real time by:

  • tracking reps
  • checking exercise form
  • structuring the workout into sets
  • progressing through exercises
  • adapting when the user hits friction

The signature feature is Workout Rescue.

If the user runs into a real-world problem such as:

  • equipment being unavailable
  • low energy
  • discomfort
  • not knowing an exercise
  • needing an easier version
  • suddenly having less time

RepWise rewrites the remaining workout and keeps the session moving.

That makes it feel less like a tracker and more like a real coach making decisions under changing conditions.

How we built it

We built RepWise as a lightweight web experience with a clear separation between workout execution, AI reasoning, and persistence.

Frontend

The frontend handles:

  • camera access
  • in-browser pose detection
  • rep counting
  • set tracking
  • workout state transitions
  • mission flow UI
  • rescue interactions
  • mission complete summary

We used pose tracking in the browser to detect supported exercises and monitor form-related signals during the live workout.

Backend

The backend is built with FastAPI and manages:

  • session creation and lifecycle
  • set / rep event handling
  • workout summaries
  • rescue logic
  • persistence

AI

We used Gemini as the intelligence layer for:

  • generating the user’s workout mission
  • adapting the workout when conditions change
  • generating the coaching-style session debrief

The project was built around the Gemini Live Agent Challenge, so the product was designed as a live, multimodal coaching experience rather than a simple text chatbot.

Google Cloud

RepWise is deployed on Google Cloud Run, and workout/session data is stored in Firestore. This allowed us to keep the deployment simple while still giving the product continuity and memory across sessions.

Product design decisions

One of the biggest shifts during development was realizing that rep counting is not the product.

Rep counting and form correction are useful, but they are increasingly expected. The more interesting problem was the friction around real workouts:

  • “What should I do today?”
  • “What if the equipment is busy?”
  • “What if I do not know this movement?”
  • “What if my energy drops halfway through?”
  • “What if I need to simplify the workout?”

That led us to redesign RepWise around a more coach-led session flow:

  1. quick setup
  2. personalized mission generation
  3. live workout guidance
  4. workout rescue
  5. coach-style debrief

This made the experience feel much more like an intelligent training companion.

Challenges we faced

1. Turning a demo into a real workout engine

It is easy to make a prototype that counts movement. It is much harder to make it feel like a real workout session.

We had to move from a simple rep counter to a proper workout engine that could track:

  • current exercise
  • set progress
  • rep progress inside a set
  • exercise transitions
  • rescue-triggered exercise replacement
  • grounded session summaries

That shift was one of the most important parts of the project.

2. Making adaptation actually useful

Generating a workout is easy. Generating a workout that still works when the user’s context changes is much harder.

We wanted RepWise to handle real situations, not ideal ones. That meant Rescue had to be more than a UI trick — it had to rewrite the remaining workout in a way that still respected the user’s goal and current constraints.

3. Designing for both beginners and intermediate users

A beginner does not need the same experience as an intermediate trainee.

Beginners need:

  • simpler exercises
  • more reassurance
  • lower-friction flows
  • fewer choices at once

Intermediate users need:

  • tighter structure
  • stronger performance framing
  • more concise coaching

Designing one product that could branch meaningfully for both was a core challenge.

4. Keeping the experience fast and fun

Because this is a live coaching product, it could not feel heavy or form-like. We had to keep the setup fast, the mission clear, and the session engaging without overbuilding the UI.

What we learned

This project taught us a few important things.

1. The best AI products remove friction, not just add features

The most valuable part of RepWise was not “AI can see your workout.” It was “AI can keep your workout from falling apart.”

That was the moment the product became useful.

2. Real-time grounding matters

Using structured workout state, rep counting, sets, and form signals made the AI outputs feel much more trustworthy. The more grounded the session engine became, the better the AI layer felt.

3. Product framing matters as much as model capability

We started with something closer to an AI gym tracker. We ended with something closer to a friction-aware live workout companion. The underlying technology mattered, but the user experience framing mattered just as much.

4. Beginner confidence is an underrated problem

A lot of fitness tools assume users already know what they are doing. Building beginner-aware flows forced us to think more carefully about confidence, simplicity, and reducing in-gym friction.

What we are proud of

The part we are most proud of is that RepWise does not stop at planning or tracking.

It can:

  • create a session for the user’s goal and context
  • track execution live
  • adapt when the workout breaks
  • finish with a grounded summary

That makes it feel like a coach that stays with you through the whole session.

What’s next

If we continued building RepWise, the next areas would be:

  • better long-term progression across sessions
  • deeper personalization from workout history
  • richer form analysis across more exercises
  • more advanced coaching for different training goals
  • stronger voice interaction and real-time guidance

But for this hackathon, we focused on building the smallest version that still feels like a real AI agent helping users train through uncertainty and friction.

Built With

Share this project:

Updates

Submission history