ImmersiFLY

This image is currently processing. Please be patient...

Inspiration

What it does

It is a system that offer pilots the opportunity engage in a more immersive and effective flight training experience. The system is in essence an affordable flight simulator that allows completion of training hours toward license certification by allowing the pilot to fly a small scale airplane under REAL CONDITIONS. The pilot is not only able to control the aircraft but is also able to feel as if they themselves are inside the cockpit of the aircraft - this is accomplished by (1) providing a live stream of the environment within the cockpit into a virtual reality headset, (2) allowing the pilot to rotate its head freely and also (3) receive feedback from the aircraft (such as roll, pitch, yaw) which is relayed by the "motion" chair.

How we built it

1) Hacked the MultiWii open source library to introduce two new protocols. One, allow the UAV to send orientation data back to a ground control device. Two, allow the UAV to receive the pilot's head orientation which is used to move the UAV's cockpit camera.

(2) Develop a Flask application which captures a video stream and send device orientation information via web socket

(2.1) Leverage OpenCV 3.3 to capture RF video transmission incoming through an RF USB recevied. (2.2) Capture deviceOrientation (pilot's head orientation) and send it using WebSockets

Challenges we ran into

(1) Thinking about the flight industry and how we could disrupt flight education through the use of technology (2) OpenCV does not support LibV4L right off the bat, so we had to recompile openCV (3) Web sockets and regular WebSockets do not have the sae protocol