Project Gym

I needed a gym coach, a muay thai trainer, a recovery plan, a nutritionist and motivation. That's 5 experts who don't talk to each other. Project Gym is one AI that knows your entire fitness journey.

Comment

The Problem That Started This

One year I was training for muscle. A few months later I wanted to try muay thai. Then I started planning for a marathon. Next up — Hyrox.

Every time I switched, I started over. New app. New trainer. No memory of what came before.

Think about what a complete fitness support system actually looks like:

  • A strength coach
  • A muay thai trainer
  • A running coach
  • A Hyrox specialist
  • A nutritionist
  • A physio for recovery
  • A mental coach for the tough days

That's 7 people managing you. And the worst part? None of them knows your full journey. They each only know a piece of it.

That's exactly what every fitness app does too. Each one wants to be the best in one domain. They forget that people don't just do one thing anymore. Nobody builds for the person who trains across multiple disciplines — and switches between them as life evolves.

I was using multiple Claude chats just to plan workouts across strength, muay thai, marathon training, and Hyrox prep. It worked — but it forgot everything between sessions. No memory. No continuity. No journey.

So I thought: what if I just built this properly?

What I Built

Project Gym is a voice-first AI fitness companion built on five pillars:

Pillar What It Means
Exercise Log any discipline by voice. Zero forms.
Diet Meals logged by voice. SEA foods understood natively.
Mood Tell it you're exhausted. It adapts. Never pushes.
Gratification Streaks, milestones, weekly reviews. Every win acknowledged.
Companionship One coach. Remembers everything. Never resets.

The core mechanic is a voice bar. Speak anything — what you trained, what you ate, how you feel, what's worrying you. Claude parses it all simultaneously, shows a review screen, and writes to the database in one action.

The Memory Architecture

This is the part I'm most proud of. Most AI fitness apps forget you after one session. Project Gym uses a three-tier hybrid memory system:

  • Tier 1 — Permanent: Injury notes, personal bests, goals, form cues. Always injected into every AI call. Never expires.
  • Tier 2 — Rolling Window: Last 30 days sessions, 14 days meals, 30 days mood. Injected fresh as structured SQL data.
  • Tier 3 — Lifetime RAG: Everything older than the rolling window, embedded into pgvector and retrieved via similarity search. Top 10 most relevant chunks per query.

Every voice log confirmation triggers an embedding job via Next.js after() — the user gets their coaching response instantly, the embedding runs silently after.

$$\text{Coach Quality} = f(\text{Permanent Context} + \text{Rolling SQL} + \text{Lifetime RAG})$$

Agentic AI — Six Agents

Project Gym doesn't just respond. It reasons and acts across multiple steps:

  1. Onboarding Intelligence Agent — Conversational voice setup. Builds full profile and first training plan in one chat session.
  2. Plan Migration Agent — User wants to go from PPL to Hyrox prep? Agent analyses current state, maps a phased transition, archives the old plan with full history.
  3. Plan Optimisation Agent — Detects plateaus and undertraining. Proposes specific changes with reasoning.
  4. Injury & Recovery Detection Agent — Detects pain language, cross-references history via RAG, proposes plan modifications.
  5. Weekly Debrief Agent — Every Sunday. Compares your week against your lifetime baseline. One story. One focus. Delivered as a notification.
  6. Re-engagement Agent — RAG-powered. References why you took breaks before and what brought you back. Never a template. Always personal.

Discipline System

Seven platform disciplines with first-class support: Strength Training · Functional Fitness (Hyrox/CrossFit) · Muay Thai / Martial Arts · Yoga & Mobility · Running & Endurance · Cycling · General / Mixed

Plus user-created disciplines — dragon boat, silat, badminton, anything. Private to the user. Claude knows what every discipline demands.

Meal Database

Three tiers — no MyFitnessPal pollution:

  • Tier 1: 200 curated SEA foods. Nasi lemak, pho, roti canai, teh tarik. Verified estimates.
  • Tier 2: Claude estimation for anything else. Always labelled "estimated."
  • Tier 3: User's personal meal library. Private. Reusable. Their numbers, their food, their trust.

How I Built It

Stack:

  • Next.js 14 App Router + TypeScript
  • Supabase Postgres + pgvector (enabled from day one)
  • Claude API — claude-sonnet-4-6 (all AI, all agents)
  • OpenAI text-embedding-3-small (vector embeddings)
  • Browser Web Speech API (voice input — no external service)
  • Framer Motion (animations and transitions)
  • next-pwa (installable on iPhone and Android — no app store)
  • Recharts (progress charts)
  • Vercel (deployment)

The entire product is a PWA. No app store. Install from browser. Works on every device in SEA.

What I Learned

1. The memory layer is the product. Voice logging is just the input. The three-tier RAG architecture is what makes the AI feel like a coach instead of a chatbot. Without lifetime memory, you have a faster form. With it, you have a companion.

2. Agentic AI changes the UX ceiling. Single-shot responses are useful. Multi-step agents that reason, query your database, and propose structured outcomes are transformative. The Plan Migration Agent — where a user describes a new goal in chat and the agent handles the full transition — is not possible with a standard chatbot.

3. Scope is a discipline. 36 hours forces brutal prioritisation. Voice bar + RAG coach chat + onboarding agent is a complete demo. Everything else is Phase 2.

4. after() is underrated. Next.js after() let me run the entire embedding pipeline asynchronously without N8n, without queues, without added infrastructure. The user gets their coaching response. The vectors write silently. Zero friction.

Challenges

Token management with three-tier injection Injecting permanent context + 30 days rolling SQL + 10 RAG chunks into every call gets large fast. The solution was aggregating the rolling window into summaries rather than raw rows — keeps the context rich but manageable.

Discipline-aware voice parsing The JSON schema needed to handle sets×reps×weight for strength AND rounds×duration for muay thai AND distance×pace for running — all from one voice note. Claude handles this naturally with the right system prompt but the schema design took iteration.

Onboarding agent conversation flow Getting Claude to ask exactly five questions — no more, no fewer — while sounding natural and not like a form, required careful prompt engineering around conversation state.

pgvector on first use The chunks table starts empty. RAG retrieval gracefully handles zero results and falls back to SQL context only — but this needed explicit handling so new users don't get degraded responses.

The Tagline

Fitness isn't just a transformation plan.

You have tough days at work and skip the gym. You switch from lifting to running to muay thai. You need someone who knows your entire story — not just your last session.

The journey matters.

That's Project Gym.

Built With

  • anthropic
  • bolt
  • claude
  • github
  • nextjs
  • openai
  • supabase
  • vercel
Share this project:

Updates