Trying out ai trainer
Wearable AI glasses
Side by side video analysis
Dashboard

GymBuddy: Real-Time AR Trainer & Adaptive AI Coach

What Inspired Us

Upon brainstorming ideas, we landed on a simple conversation about what it actually feels like to start working out. Almost everyone we talked to described the same thing. People want to be consistent, but many feel unsafe, anxious, or overwhelmed the moment they step into a gym. Public gyms, especially on university campuses, often feel like places where you need to already be an expert just to participate.

Beginners do not know where to start. Teenagers bring friends for confidence. Older adults feel pressured into expensive personal trainers. Social media adds even more confusion with advice that is often inaccurate or harmful.

At the same time, we are entering a new era of wearable technology. AR glasses, lightweight sensors, and on-device AI are becoming part of daily life. We carry powerful intelligence with us everywhere, and the idea of using that power to solve real problems felt exciting to us.

GymBuddy grew from that realization. With AR, people can get clear and reliable guidance from their very first rep. No trainer needed. No need to rely on friends. Just real support delivered through technology that feels natural to use.

What GymBuddy Does

GymBuddy is an AI-powered workout companion that helps users plan, schedule, and stay consistent with their training using a mix of conversational coaching, calendar automation, and real-time form tracking which is all powered by the GymBuddy conversational agent and computer vision.

At a high level, GymBuddy does four big things:

Understands the user’s goals and preferences
Builds and schedules workouts automatically
Adds real-time motion and form feedback during workouts
Supports accountability through conversational check-ins and reminder workflows

Core Features

1) Conversational onboarding and coaching

GymBuddy uses its conversational agent to guide users through onboarding and ongoing coaching. After signup, it naturally learns:

training goals (fat loss, strength and size, pure strength),
preferred workout days,
preferred time of day,
practical constraints like travel time to the gym.

The agent supports free-form messages so users can talk naturally instead of following rigid command syntax.

2) Real-time form and pose tracking

Using computer vision (MediaPipe + OpenCV), GymBuddy tracks key body landmarks and joint angles during exercises.
This enables immediate form feedback, safer movement awareness, and real-time corrections.

3) Automatic calendar scheduling (with conflict awareness)

GymBuddy connects to Google Calendar and schedules workouts during actual free time.
It checks existing events and avoids overlaps before creating workout blocks.
This feature turns “I should work out” into a concrete, time-boxed commitment that fits real life.

4) Dynamic updates through conversation

Users can update GymBuddy conversationally, for example:

“I can’t make it today”
“reschedule Thursday”
“I finished my workout”

GymBuddy can interpret these requests and update the plan accordingly (reschedule/cancel/mark complete), with delivery behavior depending on messaging configuration. The conversational agent interprets these messages and updates the workout plan and calendar accordingly.

5) Accountability nudges and reminders

GymBuddy sends reminders around workouts to keep users consistent:

pre-workout nudges,
post-workout check-ins,
supportive motivational follow-ups.

What We Learned

We realized quickly that GymBuddy isn’t just an AI challenge, it’s a reliability, safety, and trust challenge across the entire stack.

Safety must be instantaneous. AR form cues only help if they show up the moment a user moves incorrectly. That required fast, stable computer vision pipelines that catch risky patterns early and surface feedback before bad habits form.
Reliability is the backbone of trust. A single duplicated, delayed, or out-of-order message breaks the coaching experience. We had to treat messaging like a distributed system, with guarantees around ordering, consistency, and delivery.
State management drives user confidence. Onboarding, free-form chat, and AR guidance all need to share context so the assistant feels coherent. Any mismatch in state creates friction that can look like repeated questions, inconsistent tone, or lost progress.
Scheduling is real-world constrained optimization. Planning workouts means respecting recovery windows, gym availability, equipment changes, and a user’s actual calendar. Reliable coaching requires hard guardrails and predictable scheduling rules.
APIs introduce unpredictable complexity. Calendar sync, SMS delivery, and authentication flows behave inconsistently in real environments. These external constraints shaped our architecture just as much as CV or ML did.

How We Built It

GymBuddy is a full stack system centered on real time AR guidance and conversational coaching.

Core technologies

AR: real time form analysis, rep tracking, unsafe movement detection, subtle visual cues
Frontend: React onboarding flow with safety and experience preferences
Backend: Node.js, TypeScript, Express
Database: Prisma and PostgreSQL on Supabase
Calendar: Google Calendar API for conflict free scheduling
AI: OpenAI for natural language understanding and action mapping
Messaging: started with email-to-SMS for conversational updates, and later added voice-agent flows to improve reliability and motivation.

Core workflow

User signs up and sets experience level and safety preferences
AI agent reaches out during onboarding to build comfort
AR companion guides form rep by rep inside the gym
Scheduler chooses safe and available workout slots
If the user skips a lift or feels pain, the system adapts instantly

Challenges We Faced

1) Training Real Form Data

Collecting useful data was difficult. We filmed many videos across different body types and angles. Extracting joint angles was noisy, and finding thresholds that separated good and bad form required careful experimentation.

2) Building a Personal, Reliable AI Agent

Text messaging seemed simple, but direct SMS providers introduced compliance requirements and registration delays. We pivoted to email to SMS to move faster, but delivery latency and duplicates caused new problems.
We eventually adopted phone based voice agents. This avoided compliance bottlenecks and gave users something more valuable. A voice call can provide the motivational push that helps people follow through with a workout.

3) Real Time AR Reliability

AR guidance must be extremely low latency. Even small delays make feedback feel incorrect. We optimized our pipeline to keep guidance immediate and predictable.

4) Calendar Safety

Workout events should never overlap with existing commitments. We enforced strict conflict checks and recovery spacing before creating or modifying events.

What’s Next for GymBuddy

GymBuddy began as a proof of concept, and there is a lot more we want to build.

More Exercises and Better Coverage
Right now we support a core set of foundational movements. Our next goal is to expand the library to include a wider range of strength and functional exercises, along with more nuanced form cues and movement checks.

Integration with Real Wearable Hardware
To simulate AR during the hackathon, we taped a webcam to a pair of glasses. Our next milestone is to integrate with actual wearable hardware and lightweight AR glasses so the system feels completely natural and hands free.

A Mobile Companion App
We also plan to build a light mobile version of GymBuddy that helps users track progress, review form feedback, and manage their workout plans even when they are not wearing AR glasses.

These steps move GymBuddy from a prototype to a fully usable coaching platform that supports people at every stage of their fitness journey.

Our biggest takeaway:
A healthy fitness habit equals safety multiplied by confidence multiplied by consistency.

With AR, we believe that working out confidently, safely, and effectively can truly be as simple as putting on a pair of glasses.