OrbFit

OrbFit is the world’s first voice coach that adapts your workout as you talk. Short on time, tired, or sore? Tap the orb Just say it. OrbFit adjusts instantly and learns over time. Powered by Cartesia

Comment

Inspiration

OrbFit started because I was stuck.

I was broke, struggling in school, and not really proud of where I was heading.

At the time, I was a junior math major taking Combinatorics and a higher-level statistics course. I was doing badly enough that I eventually dropped both.

But that period changed how I thought.

In Combinatorics, I became fascinated by visualization and by the idea that sometimes the way you represent a problem matters more than the problem itself.

Around the same time, I was experimenting with memory palaces and different ways of learning.

That led me to a simple idea:

If you want to build something different, you first have to learn to think differently.

So I started changing what I spent my time on.

I entered an RNA competition on Kaggle. While I was there, I came across the AI Mathematical Olympiad – Progress Prize competition.

What caught my attention was not just the models.

It was the structure around them.

Some of the strongest systems were not simply bigger AI models. They had:

  • Instructions
  • Tools
  • Rules
  • Memory
  • Agents
  • A harness around the model

That was the moment OrbFit really started.

I remember thinking:

if you can give AI the right tools and structure to solve difficult math problems, why can't you do the same thing for someone's fitness?

And I had already experienced the problem personally.

Most fitness apps gave me a plan.

But the plan never knew what was actually happening in my life.

It didn't know that I slept badly.

It didn't know I only had 20 minutes.

It didn't know I was traveling.

It didn't know my shoulder was sore.

It didn't know what equipment I had that day.

The plan stayed the same. My life didn't.

That felt backwards.

I did not want to build another app where the user has to keep managing the software.

I wanted to build software that could manage the complexity for the user.

So the idea became:

What if the fitness app could actually listen?

What if instead of opening settings, moving sliders, changing exercises, rebuilding schedules, and constantly configuring the app, you could simply say what happened?

  • "I slept badly."
  • "I only have 30 minutes."
  • "My shoulder hurts."
  • "I'm at a hotel."

And the coach would figure out the rest.

That became the foundation of OrbFit.

There was also another bet behind it.

I believed voice was going to become a much bigger interface for software.

Typing is powerful.

But speaking is often faster.

We were already seeing people move from doing everything manually toward increasingly conversational ways of interacting with computers.

I believed consumer applications would move in the same direction.

Not because voice is new.

Because the intelligence behind voice finally became useful enough.

So about nine months ago, I started building around that idea.

Not another workout generator.

Not another calorie tracker.

Not another app that gives you a static plan.

A coach that could understand what is happening in your life and change the plan with you.

That is what became OrbFit.

What it does

OrbFit is the world's first voice coach designed to adapt your workout while you talk to it and learn more about you over time.

Other fitness apps give you a plan.

OrbFit gives you a coach.

The user can tap the orb and simply speak.

You can say:

  • "I only have 20 minutes."
  • "I slept badly."
  • "My shoulder is sore."
  • "I only have dumbbells."
  • "Make today's workout easier."

OrbFit can then use that context to adapt the workout immediately.

The goal is to bring the major parts of a user's fitness journey into one connected system.

With OrbFit, users can:

  • Build workouts through voice or chat
  • Adapt workouts based on time, fatigue, soreness, equipment, or preferences
  • Follow exercise demonstrations
  • Track completed workouts
  • Log food through voice, barcode scanning, photos, or manual entry
  • Track calories, nutrition, water, weight, and steps
  • Access strength training, stretching, mobility, and yoga exercises
  • Use health and workout data to give the coach more context over time

Most importantly, OrbFit is designed to become more useful the more the user interacts with it.

The goal is simple:

You should not have to manage your fitness app. You should be able to tell it what is happening in your life and let it figure out what to do next.

How we built it

We started with the infrastructure.

I probably could have launched faster, but I wanted OrbFit to have the foundation of a real product rather than a prototype.

We use Supabase for backend infrastructure and authentication, PostHog for product analytics so we can understand how users interact with the product, and Superwall alongside RevenueCat for paywall and subscription infrastructure. Then I built the conversational layer.

OrbFit supports both voice and text.

The voice system follows a pipeline that looks roughly like this:

Speech
  ↓
Speech-to-Text
  ↓
LLM + Agent System
  ↓
Tools + Workout Logic
  ↓
Workout Changes
  ↓
Text-to-Speech

The important part was not simply making the model talk.

It was giving the model the ability to actually do things.

That meant building a system where the coach could understand context and then use tools to:

  • Modify workouts
  • Change exercises
  • Adjust duration
  • Adapt training intensity
  • Respond to available equipment
  • Use previous user information
  • Update the plan programmatically

We also built exercise libraries across:

  • Strength training
  • Stretching
  • Mobility
  • Yoga

Then we added nutrition tracking so users would not need a completely separate calorie-tracking application.

We also integrated health and activity data so the system could eventually understand more about the user's performance, activity, and recovery.

The goal was never to add features simply to have more features.

The goal was to connect them.

The user should not have to think about the architecture underneath OrbFit.

They should just feel like the coach understands them.

Challenges we ran into

Voice is easy to demo.

Making voice feel instant is hard.

That became the hardest problem in OrbFit.

A fitness coach cannot feel like a chatbot.

If you say:

"I only have 20 minutes."

and then wait 10 seconds for something to happen, the experience is already broken.

So we became obsessed with latency.

OrbFit has to:

  1. Hear what the user says
  2. Transcribe it
  3. Understand the request
  4. Reason about the user's workout
  5. Call the correct tools
  6. Modify the workout
  7. Respond with voice

And it has to make all of that feel immediate.

Underneath a single conversation is an entire pipeline:

Speech → Transcription → Reasoning → Tools → Workout Changes → Speech

Every step adds latency.

And the obvious solution — removing intelligence to make the system faster — was not acceptable.

So we kept rebuilding the agent harness.

Again.

And again.

We changed what the model needed to think about.

We changed when tools were called.

We moved work outside the critical path.

We reduced unnecessary context.

We experimented with what could happen programmatically instead of forcing the user to wait for the model.

It became constant trial and error.

The real question became:

How do we build a brain that can adapt while keeping latency extremely low?

Another challenge was concurrency.

If OrbFit scaled to thousands of users, we could not wait until someone tapped the orb to begin creating everything needed for that conversation.

By then, we would already be too late.

So I started thinking about the problem probabilistically.

Not every active user will enter a voice session at the exact same moment.

If we could estimate the percentage of active users likely to start speaking, we could keep enough capacity available ahead of demand.

That pushed us toward a swarm-style agent architecture, where sessions and resources could be kept ready rather than forcing every interaction to start completely from zero.

As usage grows, this provisioning can happen programmatically.

The goal is to avoid this experience:

Tap Orb
↓
Wait
↓
Loading
↓
Wait
↓
Now you can speak

And instead make it:

Tap Orb
↓
Speak

That difference matters.

Then there was another problem:

The gym itself.

A fitness voice product has to work in one of the worst environments for voice.

There is:

  • Loud music
  • People talking
  • Weights dropping
  • Treadmills
  • Machines
  • Background conversations

If OrbFit interprets all of that as speech, it does not matter how intelligent the model is.

The product feels broken.

So noise cancellation became part of the product itself.

We worked on making sure the system could focus on the user rather than constantly reacting to background noise.

Then came cost.

Real-time voice can become expensive quickly.

A single interaction can involve:

Speech-to-Text
+
LLM inference
+
Agents
+
Tool calls
+
Text-to-Speech

If OrbFit was going to become something people could use every day, the economics had to work too.

I applied to the Cartesia startup program.

OrbFit was accepted, and we received access to their highest voice subscription tier for the next 12 months.

That gave us the ability to push the voice experience further while continuing to optimize the architecture underneath it.

From the outside, OrbFit looks simple.

You tap an orb.

You say:

"I only have 20 minutes."

And your workout changes.

That simplicity is intentional.

The complexity should belong to us. Not the user.

Accomplishments that we're proud of

The accomplishment I am most proud of is that OrbFit became a real product.

For roughly the last nine months, I have spent most of my time building, learning, rebuilding, and improving it.

There were periods where I was working 14–18 hours a day.

During that time, OrbFit went from an idea to a full fitness platform.

We built:

  • A real-time voice coach
  • A text-based coaching experience
  • Adaptive workout generation
  • Workout tracking
  • Nutrition tracking
  • Exercise demonstrations
  • Strength training libraries
  • Stretching and mobility libraries
  • Yoga content
  • Health and activity integrations
  • Backend infrastructure
  • Product analytics
  • Agent-based workout adaptation

What I am especially proud of is that we did not stop at generating workouts.

Generating text is easy.

Building a system that can understand context, use tools, modify a workout, remember the user, and respond through voice is much harder.

That is the system we wanted to build.

What we learned

The biggest thing I learned is that using AI well is not about finding one perfect prompt.

It is about designing the environment around the model.

The model needs:

  • Context
  • Tools
  • Constraints
  • Memory
  • Good infrastructure
  • The ability to take actions

The model itself is only one part of the product.

I also learned that the "perfect product" is a moving target.

Every time OrbFit improved, I found another way it could become:

  • Faster
  • Simpler
  • More intelligent
  • More useful
  • More natural

That process does not really end.

I also became even more convinced about voice.

The next generation of software will not be limited to buttons, forms, and menus.

More and more, people will simply tell software what they want.

And the important question will become:

Can the software actually do something useful with what you said?

That is the bet behind OrbFit.

What's next for OrbFit

OrbFit is live.

Now the focus is simple:

Get it in front of users and learn.

I know the product is not finished.

The definition of the perfect fitness coach will continue changing as we understand how people actually use OrbFit.

The next stage is about making the coach increasingly aware of the individual.

That means improving:

  • Memory
  • Training history
  • Recovery awareness
  • Nutrition context
  • Workout adaptation
  • Voice speed
  • Personalization
  • Health-data awareness

The long-term goal is for OrbFit to understand enough about your life that changing a workout feels less like configuring software and more like talking to a real coach.

You should be able to say:

"I slept badly."

"My shoulder hurts."

"I only have 20 minutes."

"I'm traveling."

And OrbFit should understand what that means for your training.

You should not have to manage your fitness app.

You should be able to tell it the truth about your life and let the coach adapt.

Built With

Share this project:

Updates

Submission history