Physical therapy is difficult, painful, and often boring in addition to the fact that it’s hard to know if you're actually doing the exercises correctly.
Enter RocketPowAR: an AR app that picks up the signals from your muscles and uses their power to fly a rocket.
Our interface guides you through where to place the sensors on your body in order to fly the rocket. The rocket's power comes from the using the electrical activity of your muscles.
We were also inspired by brain computer interfaces and the potential they offer for bringing greater accessibility to the virtual world.
What it does
The BCI device picks up electrical activity from your muscles (EMG) and sends the information to Unity which uses it as a marker to launch rockets.
The EMG markers are chosen based on physical therapy exercises to check if you are recruiting the correct muscle groups for the exercise (and to confirm that you are not using muscles you shouldn’t be).
The initial application for RocketPowAR centers around muscle rehabilitation and physical therapy. Recruiting and using the correct muscle group(s) during an exercise is a crucial part of physical therapy to strengthen weak areas and support injured structures.
Current physical therapy devices such as TENS or e-Stim use electrical signals either to bypass a self directed recruitment (you choosing to turn a muscle “on”) by sending an electrical current through electrodes and into the muscle to stimulate nerve pathways. This is incredibly unhelpful as users can’t develop an understanding of how to recruit the muscle themselves, so the moment they stop using the TENS/e-Stim device (TENS/e-Stim sessions last 10-20 minutes maybe 1/weekly), they cannot do the rehab.
These devices are cumbersome to set up, difficult to use, and the at-home kits do not support at-home users. We wanted to fill in the gaps to help. So many people live with pain that they brush off as "a part of getting old". It does not have to be that way.
How we built it
We used electrodes to get a muscle signal to the OpenBCI Ganglion board. This board acted as a signal amplifier and used a Simplee BLE Radio module. The signals were picked up over bluetooth by using a node.js library and creating a local HTP server which the Magic Leap One client receives. We processed the appropriate live signal and chose a threshold for when a muscle would register as passively active, to being “recruited” for the exercise within the node scripts and any signals higher than threshold were used as a marker for “go” by Unity to instantiate the rocket thrusters. Once received by Unity, the animation Magic happens and our AR rocket is launched.
To sum up: Muscle electrical activity is sent to the Ganglion which amplifies the signal, the signal is then sent and parsed, it’s then sent again which extracts out if the muscle activation meets a set threshold, at that point it’s then sent to Unity which accepts it as a “go” signal to fire up the rocket’s engines, which sends it to the Magic Leap and the rocket is poweARd.
Challenges we ran into
Oh the bluetooth issues you'll have
We needed to use 5 pieces of software to get from the chip to Unity (and then Magic Leap). Of those - on Windows bluetooth wouldn’t connect, the OpenBCI software wouldn’t open (and when we found a work around it wouldn’t pick up the Ganglion). The signal processes software only runs on Windows. Running a virtual machine on the Mac broke the Mac (oops). We then tried using docker to run windows on Mac. Same issues as above. Then we installed Unbuntu onto a external disk…but if the laptop was ever turned off or shutdown then we would lose everything. Eventually we thought to use web sockets except that the documentation on that was very very very misinformed.
So now the Ganglion is connected. But what is the data coming in. We had “random” (what we thought was voltage) coming in from channels that weren’t being used in the form of a 4 size array. And that data was not related in any way to muscle activity. So after both hardware and software debugging, we found a working channel, we found the actual voltage information, and extracted the signal which was the muscle contraction.
Did we mention that we had no programs to interpret or push the signal? Thank you mentors for a team crash (literally) course in terminal
Sending data end to end was a massive massive challenge
Bluetooth would dropped every 4 seconds
Quote from a mentor: “learning how to use OpenBCI is it’s own masters degree and you need a PhD just to understand the documentation” (all of which was out of date).
None of us had any experience with OpenBCI or EMG signals before, only one of us had ever worked with Magic Leap, and we had just enough to keep us going. Our “wins” were:
Picking up heart rate signal (ECG) using Ganglion in ECG specific software built for OpenBCI
“Bluetooth pairing successful”
Pick up a signal and have it display in terminal
Stream —> live
Getting a single bit sent end to end (Ganglion to Magic Leap)
Finding a a channel that sends signals
Parsing the signal into something that could be understood
What we learned
How to use OpenBCI (and why correct documentation is important) How to process and use biometric data How to send biometric data using websockets Developing for the Magic Leap Using terminal Everything related to bluetooth Networking
What’s next for RocketPowAR
Adding in additional physical therapy exercises for each injury (fly the rocket through the levels of the stratosphere, avoid meteors), and then adding in different "missions" based on different exercises (e.g. strengthening up VMO after a ACL or MCL tear). We want to find a better way to work with the EMG signals, and can spend more time looking into signal processing software. We also want to spend more time on the send along of the data instead of having to use sockets.