In The Avengers(2012), Tony Stark is recruited into the Avengers initiative. Upon being recruited, he is handed a file containing the details of the Avengers' mission. Tony Stark expands the file into a large hologram and review it. I'm not a fan of Marvel, but in this scene, Tony Stark is able to pick up a hologram with his hand. I wanted to recreate this ability in XR.

What it does

Right now, this is a port of Google's MediaPipe Hand Detector for ThreeJS. It finds your hand, transforms the coordinates of your hand into ThreeJS coordinates(as if you don't the hand doesn't look right) and draws it on a web browser.

How we built it

The tool is implemented in JavaScript. It uses ThreeJS to draw objects on the web browser canvas. It also partially implements a physics engine(Cannon.JS) that allows the hand to manipulate objects in AR. The first step to building was to investigate EchoAR, which was the tool I wanted to use to render objects in Augmented Reality. While EchoAR allows you to spawn premade objects in AR, it quickly became apparent that there wasn't a good way to actually develop scenes for WebXR using EchoAR. It only allows you to bring up already loaded objects into WebXR. You can also edit metadata to manipulate objects, but this seemed to not gel too well with Cannon.JS or ThreeJS. Then, I created a demo that allowed my hand to be detected by MediaPipe, and began building from that. I first implemented the physics engine, starting with using Ammo.JS. But Ammo.JS doesn't work with ThreeJS, so after several hours of trying, I had to switch to Cannon.JS. At this point I combined the Cannon.JS/Three.JS scene into the current product. Finally I added the ability to manipulate an object with your pointer finger!

Challenges we ran into

Many of the tools in this domain don't go well together and there's limited hardware support for XR. For example, intially MediaPipe could not work with ThreeJS because they each used two separate WebGL contexts. In order to get the two to work together, I had to switch MediaPipe to work on a WebGL context, and remake the rendering of the hand. Another example of tools not going well together is Unity, EchoAR, and WebGL. I don't have an Andoird Phone, and on a PC you can't build for iPhone, so I was forced to use WebXR to render my scene. But EchoAR didn't have any documentation for going from Unity to WebXR; only Unity to a Mobile Device. This does make sense, as it makes no sense for a desktop computer to have AR, but it forced me to not use Unity.

Another challenge in this problem that I ran into was that the depth estimation for the hand is always 0. This prevents the software from functioning correctly unless an object is placed at (0,0,0). This isn't ideal because it means that an object can only be rotated in place; it can't be moved. The way to solve this problem would be to use a neural network. NYU made a neural network that performs well for this problem. However, this wasn't added to this project because the hand detector already had a low framerate(around 20 FPS) and I didn't want to decrease it further to ensure that the submission is usable.

Accomplishments that we're proud of

  • The hand appears in XR.
  • The scene is physics based
  • Its not a far cry to get the hand to work with Cannon.JS
  • The scene is accessible on all devices. ## What we learned
  • Don't do hackathons alone. There is a lot of work to be done and its best to split the work among a group.
  • Hackathon projects should be small to fit in the time frame, if they are too large, they won't be polished.
  • It is much easier to simulate Human poses that I previously thought.
  • I learned how to use ThreeJS and WebGL.
  • I improved my JavaScript skills
  • The reason that physical interaction in XR is rare because of the monocular depth estimation problem, which is usually solved with neural networks, which are computationally expensive. ## What's next for Hands in XR The next milestone is to put the hand into Cannon.JS. This means that each "bone" in the hand must be transformed into a co indecent box and that box must be attached to a RigidBody in Cannon.JS. After that, making the hand work in AR must be done. ThreeJS supports AR, so this will not be too difficult once Hands in XR can be hosted on something like Google Cloud. After that, Hands in XR will allow physics-based interactions with 3D models in XR.


I got it working with echoAR! Now you can manipulate an echoAR object with your hand in 2 dimensions.

Built With

Share this project: