FitFlow – Hand-Gesture MR Workout Coach

Inspiration

Exercising in VR is powerful, but many fitness apps rely on bulky avatars or fully virtual environments that reduce awareness of the real world. We wanted to explore whether simple hand gestures and mixed reality overlays could create a safer, lighter, and more accessible workout experience on Meta Quest.

FitFlow was inspired by the idea of turning your real room into a guided workout space—without controllers, without headsets blocking awareness, and without complex 3D characters.

What it does

FitFlow is a hand-gesture based mixed-reality workout coach for Meta Quest. Using passthrough mode, it overlays clean UI elements such as rep counters, timers, and posture hints directly onto the user’s real environment.

The app tracks basic hand movement using Quest’s hand tracking to detect exercise repetitions in real time. During a session, users can see:

• Live rep counting
• Session timer
• Simple posture feedback (e.g. “Raise higher”)
• A summary screen with total reps and duration

The goal is to keep workouts simple, visual, and safe.

How we built it

FitFlow is built as a native Android app for Meta Quest using Kotlin and Jetpack Compose. Hand joint data from Meta Quest’s hand tracking APIs is processed using lightweight heuristics and a small state machine to detect full movement cycles (up–down motion) with debouncing and smoothing to avoid false counts.

Instead of complex avatars or heavy AI models, we intentionally focused on:

• Minimal UI overlays
• Passthrough-first design
• Reliable, explainable rep detection logic This approach allowed us to prototype quickly while keeping performance high.

Challenges

The biggest challenge was handling hand tracking noise and avoiding false rep counts. We solved this by smoothing joint data and enforcing thresholds and timing constraints before counting a repetition.

Another challenge was designing UI that remains readable without obstructing the real world, which we addressed through transparent overlays and large, high-contrast elements.

What we learned

We learned that even simple hand movement data can unlock powerful mixed-reality experiences when paired with thoughtful UX. Passthrough combined with hand interactions creates strong immersion without sacrificing safety.

Future improvements

Future versions of FitFlow could add support for more exercises, improved posture feedback using on-device ML, voice coaching, workout history, and social or challenge-based modes.

Built With

• android
• hand-tracking
• jetpack-compose
• kotlin
• meta-quest
• passthrough