In The Avengers(2012), Tony Stark is recruited into the Avengers initiative. Upon being recruited, he is handed a file containing the details of the Avengers' mission. Tony Stark expands the file into a large hologram and review it. I'm not a fan of Marvel, but in this scene, Tony Stark is able to pick up a hologram with his hand. I wanted to recreate this ability in XR.
What it does
Right now, this is a port of Google's MediaPipe Hand Detector for ThreeJS. It finds your hand, transforms the coordinates of your hand into ThreeJS coordinates(as if you don't the hand doesn't look right) and draws it on a web browser.
How we built it
Challenges we ran into
Many of the tools in this domain don't go well together and there's limited hardware support for XR. For example, intially MediaPipe could not work with ThreeJS because they each used two separate WebGL contexts. In order to get the two to work together, I had to switch MediaPipe to work on a WebGL context, and remake the rendering of the hand. Another example of tools not going well together is Unity, EchoAR, and WebGL. I don't have an Andoird Phone, and on a PC you can't build for iPhone, so I was forced to use WebXR to render my scene. But EchoAR didn't have any documentation for going from Unity to WebXR; only Unity to a Mobile Device. This does make sense, as it makes no sense for a desktop computer to have AR, but it forced me to not use Unity.
Another challenge in this problem that I ran into was that the depth estimation for the hand is always 0. This prevents the software from functioning correctly unless an object is placed at (0,0,0). This isn't ideal because it means that an object can only be rotated in place; it can't be moved. The way to solve this problem would be to use a neural network. NYU made a neural network that performs well for this problem. However, this wasn't added to this project because the hand detector already had a low framerate(around 20 FPS) and I didn't want to decrease it further to ensure that the submission is usable.
Accomplishments that we're proud of
- The hand appears in XR.
- The scene is physics based
- Its not a far cry to get the hand to work with Cannon.JS
- The scene is accessible on all devices. ## What we learned
- Don't do hackathons alone. There is a lot of work to be done and its best to split the work among a group.
- Hackathon projects should be small to fit in the time frame, if they are too large, they won't be polished.
- It is much easier to simulate Human poses that I previously thought.
- I learned how to use ThreeJS and WebGL.
- The reason that physical interaction in XR is rare because of the monocular depth estimation problem, which is usually solved with neural networks, which are computationally expensive. ## What's next for Hands in XR The next milestone is to put the hand into Cannon.JS. This means that each "bone" in the hand must be transformed into a co indecent box and that box must be attached to a RigidBody in Cannon.JS. After that, making the hand work in AR must be done. ThreeJS supports AR, so this will not be too difficult once Hands in XR can be hosted on something like Google Cloud. After that, Hands in XR will allow physics-based interactions with 3D models in XR.
I got it working with echoAR! Now you can manipulate an echoAR object with your hand in 2 dimensions.