ControlAR is an augmented reality tool built for the Magic Leap that allows the user to turn everyday objects into controllers that affect in-game objects.

What it does

ControlAR is diverse in its applications and can be applied to many different fields. In our demo, you can fly a plane around the room by simply pointing a banana in the direction you want the airplane to fly. As of now, this is done using the gyroscope inside a phone.

How we built it

This initial version of ControlAR is made with Unity, written in C#, and deployed to Magic Leap. The Unity client can also run on Android, allowing it to be used as an alternate controller. Camera frames (YUV data in byte buffers) are streamed via UDP to a custom Python server we wrote, which is intended to as a bridge between the Magic Leap 1 and the deep learning runtime. We were able to build and configure PoseCNN (an open-source pose estimation framework implemented from a research paper) from source and run basic images through it on an Nvidia GPU with CUDA support. We started work on converting YUV frame buffers to RGB data that is compatible with PoseCNN, but ran out of time for integration toward the end of the hackathon. We intend to finish the PoseCNN integration at some point in the near future.

Challenges we ran into

  • Linux OS
  • Raspberry Pi
  • Deep learning framework - CUDA
  • Magic Leap OpenCV

Accomplishments that we're proud of

  • Streaming to/from a remote server for CUDA-based PoseCNN
  • Magic Leap deployment and feature exploration
  • Cohesive teamwork

What we learned

  • Networked streaming of camera data
  • Convolutional neural network integration
  • Have GPU environments set up ahead of a hackathon for deep learning, just in case

What's next for ControlAR

As aforementioned, ControlAR is a tool that can be used for a diverse array of applications. As we continue to implement machine learning, we plan to continue to optimize ControlAR, while adding more minigames showcasing potential functionalities.

Built With

Share this project:

Updates