Android landing screen
Hand overlaid in AR
Fist recognition from Myo
Desktop landing screen
Phantom limb syndrome is a condition in which amputees feel a sensation in their missing limb, resulting in immense pain. Although this pain is not actual physical pain, current treatments include medication with negative side effects and invasive brain-computer interfaces. One treatment that has been used as well is the "box and mirror" method, in which the amputee reflects off a mirror to make it seem like the limb is no longer missing. However, this method is largely unrealistic and takes about 30 weeks to relieve the pain. ThARapy seeks to change that.
What it does
ThARapy is an Augmented Reality simulation that alleviates phantom pain but using EMG data from the Myo Armband to control a virtual hand. When a hand is amputated, the muscles that control the hand (in the arm) are actually still functional; they simply don't have anything to control. This is another contributor to phantom limb syndrome-- every time those muscles contract, there is more pain. ThARapy seeks to solve this problem by creating an interface where amputees can actually feel like their missing limb is back, and provides an additional layer of interaction by allowing them to control the hand using existing muscles. The Myo Armband sends EMG data to the app, while Vuforia powers the AR system that brings it to life.
Another feature of thARapy that sets it apart from outdated methods is that it provides analysis and data visualization for the user's progress over time. When the user goes through the simulation, they answer a series of questions that let the app & the physician know how they are doing. The data collected from the simulations gets stored on the user's device and is displayed in the Progress screen in a user-friendly manner (with UX in mind).
How I built it
I built the app as an Android app (with cross-platform support) using the Unity3D game engine. I used Vuforia for computer vision, so the camera detects an MLH sticker as the visual marker for where the hand should start. Then, a collection of UI elements and C# scripts runs through the simulation as the app reads EMG data inputted through the Myo Armband. Lastly, the data is stored locally and displayed using dynamically generated graphs and statistics.
Challenges I ran into
Initially, I wanted to create thARapy for the Microsoft Hololens so that users would not have to hold up their phone or use a computer. However, I only had a Mac computer with me and I was working alone, so after trying for a while to get the Hololens working, I gave up and made it an Android/desktop app. The biggest challenge was definitely getting the Myo Armband SDK to work on Android phones. Myo currently has no SDK or plugin for using Myo with Android in Unity, so I actually had to create my own plugin to make it work. This took hours to do, and I experienced a LOT of issues with Bluetooth Low Energy connection.
Accomplishments that I'm proud of
Working alone. This was a daunting project to take on without a team, and I am proud of making a functional app with a nice user interface.
What I learned
Augmented Reality. I have never worked with mobile AR before, and I am happy to have learned an important skill during HackGT. I will definitely make more AR in the future. I also learned how to work with the Myo Armband, which I have never worked with before.
What's next for thARapy
I would like to add more gestures to the system and make it even more accurate. I also want to incorporate a heart rate sensor to track pain and incorporate it into the Hololens. I would have used the heart rate sensor in my app this weekend, but there were none at the hardware checkout.
Overall, thARapy is an efficient, low-cost at-home treatment and management system for phantom pains using Augmented Reality.