Can someone create an _ affordable _ VR headset capable of augmented reality on a drone? SURE!!! Can be done with an affordable off the shelf drone? SURE!!!

Can we make an environment where both real and virtual entities can cooperate to give an enhanced experience? SURE!!

What it does

We used an off the shelf parrot 2.0 drone and used its WiFi signal to both control the drone via an interactive VR control (leap motion) and use the stream for a android app that has Google Cardboard embedded onto it and also a renderer that uses Open CV (a computer vision library) to detect a pattern and produce a rendered object.


  • Phillipe Clifton
  • Loren Isakson
  • Arturo Santa
  • David Solano

Challenges we ran into

  • We had a 40 second delay on the FFMPEG right after its creation!!!! (Solved)
  • We had to capture a simple WiFi signal and split it for _ video _ and _ controls _ on two separate machines (Solved)
  • Tried to use the cookies to export weather data from the drone to a phone (Partially solved, several problems of payload and connectivity, but we managed to send a stream of temperature via Bluetooth. We will use it as a weather station in the stand)
  • Incompatibility between a reduced version of OpenGL on phones (GLES 2) and most Computer vision libraries (OpenGL 1 mostly) (Worked around it by directly drawing the figures, very simple shapes like a gray "building")
  • 3D printing of a ball and socket joint takes a long time compared to the length of the hackathlon (Solved, we got it working)
  • Finding some wheels for the base of the mount at 4:30 A.M (Solved, destroyed a mop bucket)

Accomplishments that we are proud of

  • Created an app that combines GoogleCardboard, FFMPEG streaming, and OpenGL
  • We are able to obtain the video-feedback from the AR-drone and send it to our developed app that shows the augmented reality environment while controlling the quadcopter via hand motions through the Leap-motion.
  • Successfully hacked and implemented two WiFi signals using different ports to the quad rotor.
  • Successfully hacked the Leap-motion thus obtaining and transmitting hand gestures in real time.
  • Designed and built a test-stand in which to experiment with future quadcopter concepts
  • 3D printed the customized the main hinge / ball bearing of the stand mechanism by exporting the file created in Solidworks to an STL file.
  • Laser cut the other assembly parts.

What we learned

  • How to compile a bunch of libraries together in a record time!!!
  • How cool Node js really is
  • How hard is to mix Google Cardboard and FFMPEG
  • How to do FAST 3D printing and Laser Cutting

What's next for Drone Augmented Reality and Advanced Motion Controls

  • Increasing the number of objects that it can render by training more complex patterns
  • Compile an instance of OpenOSG in GLES 2.0 to actually use pre-rendered objects
  • Increase the resolution of streaming by improving the signal, cameras and cellphones

Built With

Share this project: