Inspiration
Rehabilitation exercises can often become repetitive and difficult to maintain consistently. We wanted to explore how Mixed Reality (MR) could make rehabilitation more interactive, motivating, and measurable. This led us to develop an MR rehabilitation experience for upper-limb and hand exercises, using Meta Quest devices with hand tracking and passthrough technology. Our goal was to create an environment where patients can perform guided exercises while receiving visual and interactive feedback. The project focuses on applications such as hand rehabilitation, post-stroke therapy, neurological rehabilitation, orthopaedic recovery, and occupational therapy. MR_Rehabilitation_3D_Immersive_…
What it does
Our platform transforms conventional rehabilitation exercises into interactive Mixed Reality experiences. Using Meta Quest devices, patients can perform guided hand and upper-limb exercises through natural hand tracking and passthrough MR. A 3D avatar and interactive visual/audio feedback guide users through progressive exercises, from light pressure control to stronger grip activities. The platform is designed to support rehabilitation scenarios including hand therapy, post-stroke recovery, neurological rehabilitation, orthopaedic recovery, and occupational therapy. Testimonial-MR Rehabilitation The planned platform architecture connects patient interaction, AI guidance, performance analytics, cloud data, and therapist dashboards.
How we built it
We built the prototype using Unity 3D, Meta Quest MR capabilities, hand tracking, passthrough, 3D assets, and interactive XR technologies. The technical architecture also incorporates OpenXR, XR Interaction Toolkit, Meta SDK, Blender, APIs, REST services, and cloud storage. MR_Rehabilitation_3D_Immersive_… The experience uses natural hand gestures such as pinch and grasp, combined with visual guidance, audio feedback, and a 3D avatar. Testimonial-MR Rehabilitation
Challenges we ran into
One of our major challenges was designing natural and reliable hand interactions while keeping exercises simple enough for users to understand and perform. We also had to consider progressive difficulty, clear feedback, device performance, and usability in both clinical and home environments. Another challenge was designing the system so that rehabilitation activities could eventually generate meaningful performance data for therapists rather than remaining purely gamified experiences. Our current project is a technology prototype and development platform. Further clinical validation, standardized assessment protocols, and clinical evaluation would be required before making clinical efficacy claims.
Accomplishments that we're proud of
We are proud to have transformed a conventional hand exercise concept into an interactive MR rehabilitation experience where users can interact naturally with virtual elements using their hands. Key accomplishments include:
- Implementing MR passthrough and hand tracking.
- Developing progressive rehabilitation exercises.
- Creating a 3D avatar-based guidance system.
- Adding motivational audio and visual feedback.
- Designing the experience for potential use in both clinical and home environments. Testimonial-MR Rehabilitation
- Establishing a foundation for future AI guidance and therapist analytics.
What we learned
During development, we learned that effective XR rehabilitation requires more than simply creating an immersive 3D environment. Interaction design, exercise progression, clear visual feedback, natural hand tracking, motivation, and performance measurement all need to work together. We also learned how to combine Unity-based XR development with hand tracking, 3D interaction, AI integration, and analytics to create a foundation for more personalized rehabilitation experiences.
What's next for Mixed Reality technologies in Rehabilitation
We see MR evolving from simple exercise applications into intelligent, personalized rehabilitation platforms. Our next direction is to integrate AI-based coaching, adaptive exercise difficulty, performance analytics, and therapist dashboards so that rehabilitation experiences can respond to individual user progress. We also envision expanding beyond hand rehabilitation into stroke recovery, neurological rehabilitation, orthopaedic therapy, occupational therapy, and other upper-limb rehabilitation scenarios. MR_Rehabilitation_3D_Immersive_… The long-term vision is a connected ecosystem where MR provides the immersive therapy environment, AI provides personalized guidance, and analytics provide measurable progress insights for therapists.
Log in or sign up for Devpost to join the conversation.