I love the GearVR, but a VR experience can be drastically improved with hand tracking/input. When I heard PMD was providing Pico Flexx dev kits it seemed like the perfect opportunity to try integrating hand tracking into the GearVR.
What it does
It allows users to track their hands and send the data to the GearVR. As a proof of concept I made a quick simple app that is similar to Google's Tiltbrush on the HTC Vive, which let's you draw in 3-Dimensions in VR.
How I built it
I had to first get the data off the Pico Flexx and into Unity and then use the Unity networking framework to transmit the hand data to the GearVR. The GearVR was running it's own version of the experience which simply listened to commands from the PC which was doing all the hand tracking.
Challenges I ran into
It took me a while to get the unity networking off the ground. I also had to get familiar with the Carnival SDK and seeing what was possible with the PIco Flexx. I tried sending the entire point cloud data via the network, but it didn't seem efficient for bandwidth so I just went with hand position.
Also, the Pico Flexx seems to be optimized for top down viewing using the Cardboard holder provided for hacking. So I had to mirror all the data coming out to use it from a bottom up perspective.
Accomplishments that I'm proud of
Took the Pico Flexx hacking a step further with a headset that provides a better VR experience than cardboard.
What I learned
What's next for GearVR Hand Tracking
The drawing app I made to showcase the hack is only a starting point for what you can do. I was also thinking about how the Pico Flexx could be potentially used for positional tracking although the lag isn't ideal. But it's exciting to think of hand tracking with mobile VR to help people further extend what they can already do with their cell phone.