Conventional Rehabilitation, is often based on the repetition of spcified movements. This can result in a low engagement and in a poor compliance by the patient. There are many techonogical solutions to make it more engaging and interesting, but they have still some limitations. Among the many, two technologies are currently largely studied: Kinect based rehab and VR/AR based rehabilitation.
The first has great potentials, but it is not immersive and forces the patient to interact with an avatar in a virtual environment presented usually on a large screen. on the other hand, Immersive VR/AR (usually based on Head Mounted Displays (HMD), can be higly immersive, but they require other devices to keep track of the whole body segments or they are just limited to upper limbs or solely the tracking of the hands (when they are in the line of sight of the sensor).
We thought to integrate Kinect and Hololens 2 to combine the best features of both technology, offering the patient an immersive (and engaging) experience, while tracking (and measuring) all the ody segments, included those not tracked by the Hololens2
What it does
It offers a interesting and interactive rehabilitation experience, based on a simple nice story of a possible aline invasion that the patient should fight, by playin two different games, one stationary, more relate to upper limbs motor activities, and one more cogintive and full-body exercise
How we built it
We used Unity3d/MRTK and C# to develop two app, one which runs on the PC where the Kinect for Azure is connected, and one directly on the Hololens
Challenges we ran into
Make Kinect and Hololens talk to each other CAlculate the transformation from the Kinect reference system into the Hololens one
Accomplishments that we're proud of
We succesfully developed a nice and engaging environment for a funny and engagin AR Rehabilitation experiece, based on the idea of a small town where to fight an alien invasion.
What we learned
We learnt a lot about the different approaches of the 4 members of the team and how to deal with these brand new devices.
What's next for ARehab
Hopefully we'd like to make it an extended version more stable and robust.