Inspiration
I was inspired by the idea of making fitness more accessible and natural in VR. Many apps rely heavily on controllers, but I wanted to explore how hand tracking and passthrough overlays could guide users through yoga and workout sessions in a way that feels intuitive and immersive.
What it does
AURA FIT is an immersive fitness app for Meta Horizon OS that guides users through yoga and workout sessions using natural hand interactions instead of controllers. It blends passthrough overlays with spatial audio to correct posture, provide real-time feedback, and create calming environments. The app makes fitness accessible, interactive, and social — whether you’re practicing alone or joining a live instructor-led class.
How I built it
We developed AURA FIT in Unity 2022 with Meta XR SDK v81+ for hand tracking and gesture recognition. The Spatial SDK v0.8.0+ was used to anchor passthrough overlays in real-world space, while lightweight ML models (TensorFlow Lite/ONNX) powered pose estimation and correction. Cloud services like Firebase handled progress tracking and multiplayer sessions. We tested flows using the Meta Quest Developer Hub emulator and validated gestures with mock inputs before optimizing for Quest devices.
Challenges I ran into
Developing without direct access to Quest 3 hardware meant relying on emulation and Quest 2 compatibility. Ensuring hand tracking accuracy in varied lighting conditions. Balancing performance with immersive overlays to keep latency low. Writing a clear privacy policy for passthrough camera usage.
Accomplishments that I'm proud of
Created a controller-free fitness experience that feels intuitive and immersive. Successfully integrated pose correction overlays using passthrough and AI. Designed accessible features like voice prompts, captions, and simplified gestures. Built modular workout templates that can scale into yoga, HIIT, and meditation flows. Achieved smooth performance in emulator testing with low latency hand tracking.
What I learned
How to leverage Meta XR SDK hand tracking APIs for natural interactions. The importance of accessibility in VR fitness design. How to simulate passthrough overlays without Quest 3 hardware. Best practices for balancing immersion with performance optimization. How social and multiplayer features can increase user motivation and retention.
What's next for AURA FIT
Expand workout libraries with more routines and difficulty levels. Add multiplayer social classes with live instructor avatars. Integrate wearable health data (heart rate, calories) for personalized feedback. Optimize for Quest 3 passthrough fidelity once hardware access is available. Explore monetization through premium guided sessions and school/educational fitness programs.
How does it works
- Real-time Gesture Recognition: The AI will continuously monitor the video feed for a "Thumbs Up" gesture to automatically transition to the next exercise.
- Live Coaching: Instead of static text, an AI instructor will verbally guide you through the workout, correcting form and offering encouragement in real-time.
- Hands-Free Experience: You can complete an entire session without touching the screen.
Log in or sign up for Devpost to join the conversation.